Skip to content

Commit bcf8f01

Browse files
mythimythi
authored
Merge pull request #1512 from hj-johannes-lee/PR-2023-027
e2e: make running subsets of e2e tests more organized
2 parents b38141d + 5347665 commit bcf8f01

File tree

11 files changed

+196
-54
lines changed

11 files changed

+196
-54
lines changed

‎.github/workflows/lib-e2e.yaml

+2
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,11 @@ jobs:
3434
runner: simics-gnr
3535
images: intel-iaa-plugin intel-idxd-config-initcontainer accel-config-demo intel-deviceplugin-operator
3636
- name: e2e-qat
37+
targetjob: e2e-qat FOCUS=Resource:generic
3738
runner: qat
3839
images: intel-qat-plugin intel-qat-initcontainer crypto-perf
3940
- name: e2e-qat4
41+
targetjob: e2e-qat FOCUS="Mode:dpdk.*Resource:(cy|dc)" SKIP=App:crypto-perf
4042
runner: simics-spr
4143
images: intel-qat-plugin intel-qat-initcontainer openssl-qat-engine
4244
- name: e2e-sgx

‎DEVEL.md

+82-9
Original file line numberDiff line numberDiff line change
@@ -183,34 +183,107 @@ container images with the executables under test must be available in the
183183
cluster. If these two conditions are satisfied, run the tests with:
184184

185185
```bash
186-
$ go test -v ./test/e2e/...
186+
# Run all e2e tests in this repository
187+
go test -v ./test/e2e/...
187188
```
188189

189-
In case you want to run only certain tests, e.g., QAT ones, run:
190+
If you need to specify paths to your custom `kubeconfig` containing
191+
embedded authentication info then add the `-kubeconfig` argument:
190192

191193
```bash
192-
$ go test -v ./test/e2e/... -args -ginkgo.focus "QAT"
194+
go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig
193195
```
194196

195-
If you need to specify paths to your custom `kubeconfig` containing
196-
embedded authentication info then add the `-kubeconfig` argument:
197+
The full list of available options can be obtained with:
197198

198199
```bash
199-
$ go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig
200+
go test ./test/e2e/... -args -help
200201
```
201202

202-
The full list of available options can be obtained with:
203+
In most cases, it would not be possible to run all E2E tests in one system.
204+
For running a subset of tests, there are labels that you can use to pick out specific parts.
205+
You can run the tests with:
206+
```bash
207+
# Run a subset of tests
208+
go test -v ./test/e2e/... -args -ginkgo.focus <labels in regex> -ginkgo.skip <labels in regex>
209+
```
210+
211+
#### Table of Labels
212+
213+
| Device | Mode | Resource | App |
214+
|:-------|:-----------------|:------------|:-------------------------------|
215+
| `dlb` |- | `pf`, `vf` | `libdlb` |
216+
| `dsa` |- | `dedicated` | `accel-config` |
217+
| `fpga` | `af`, `region` | | `opae-nlb-demo` |
218+
| `gpu` |- | `i915` | `busybox`, `tensorflow` |
219+
| `iaa` |- | `dedicated` | `accel-config` |
220+
| `qat` | `dpdk` | `dc` | `openssl` |
221+
| `qat` | `dpdk` | `cy` | `openssl`, `crypto-perf` |
222+
| `qat` | `dpdk` | `generic` | `crypto-perf`, `compress-perf` |
223+
| `qat` | `kernel` | `cy1_dc0` | `busybox` |
224+
| `sgx` |- | | `sgx-sdk-demo` |
225+
226+
#### Examples
227+
228+
```bash
229+
# DLB for VF resource without any app running
230+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:dlb.*Resource:vf.*App:noapp"
231+
232+
# FPGA with af mode with opae-nlb-demo app running
233+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:fpga.*Mode:af.*App:opae-nlb-demo"
234+
235+
# GPU with running only tensorflow app
236+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:gpu.*App:tensorflow"
237+
#or
238+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:gpu" -ginkgo.skip "App:busybox"
239+
240+
# QAT for qat4 cy resource with openssl app running
241+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:qat.*Resource:cy.*App:openssl"
242+
243+
# QAT with dpdk mode for qat2 generic resource with all apps running
244+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:qat.*Resource:generic.*App:(crypto-perf|compress-perf)"
245+
246+
# SGX without running sgx-sdk-demo app
247+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:sgx" -ginkgo.skip "App:sgx-sdk-demo"
248+
249+
# All of Sapphire Rapids device plugins
250+
go test -v ./test/e2e/... -args -ginkgo.focus "Device:(dlb|dsa|iaa|qat|sgx)"
251+
```
252+
253+
## Predefined E2E Tests
254+
255+
It is possible to run predefined e2e tests with:
256+
```
257+
make e2e-<device> [E2E_LEVEL={basic|full}] [FOCUS=<labels in regex>] [SKIP=<labels in regex>]
258+
```
259+
260+
| `E2E_LEVEL` | Equivalent `FOCUS` or `SKIP` | Explanation |
261+
:-------------- |:---------------------------- |:------------------------------------------------------------------------------------------------ |
262+
| `basic` | `FOCUS=App:noapp` | `basic` does not run any app pod, but checks if the plugin works and the resources are available |
263+
| `full` | `SKIP=App:noapp` | `full` checks all resources, runs all apps except the spec kept for no app running |
264+
265+
### Examples
203266

204267
```bash
205-
$ go test ./test/e2e/... -args -help
268+
# DLB for both of pf and vf resources with running libdlb app
269+
make e2e-dlb E2E_LEVEL=full
270+
271+
# QAT for cy resource with running only openssl app
272+
make e2e-qat FOCUS=Resource:cy.*App:openssl
273+
274+
# QAT for dc resource without running any app
275+
make e2e-qat E2E_LEVEL=basic FOCUS=Resource:dc
276+
277+
# GPU without running tensorflow app
278+
make e2e-gpu E2E_LEVEL=full SKIP=tensorflow
206279
```
207280

208281
It is also possible to run the tests which don't depend on hardware
209282
without a pre-configured Kubernetes cluster. Just make sure you have
210283
[Kind](https://kind.sigs.k8s.io/) installed on your host and run:
211284

212285
```
213-
$ make test-with-kind
286+
make test-with-kind
214287
```
215288

216289
### Run Controller Tests with a Local Control Plane

‎Makefile

+17-10
Original file line numberDiff line numberDiff line change
@@ -140,29 +140,36 @@ REG?=$(ORG)/
140140
TAG?=devel
141141
export TAG
142142

143+
ifeq ($(E2E_LEVEL), $(filter $(E2E_LEVEL), full))
144+
GENERATED_SKIP_OPT=-ginkgo.skip "App:noapp"
145+
else ifeq ($(E2E_LEVEL),basic)
146+
ADDITIONAL_FOCUS_REGEX=App:noapp
147+
else
148+
$(error Unsupported E2E_LEVEL value: $(E2E_LEVEL). Possible options: full, basic)
149+
endif
150+
GENERATED_SKIP_OPT += $(if $(SKIP),-ginkgo.skip "$(SKIP)")
151+
ADDITIONAL_FOCUS_REGEX := $(if $(FOCUS),$(FOCUS).*$(ADDITIONAL_FOCUS_REGEX),$(ADDITIONAL_FOCUS_REGEX))
152+
143153
e2e-fpga:
144-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "FPGA" -delete-namespace-on-failure=false
154+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:fpga.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
145155

146156
e2e-qat:
147-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "QAT Gen2" -delete-namespace-on-failure=false
148-
149-
e2e-qat4:
150-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "QAT Gen4" -ginkgo.skip "dpdk crypto-perf" -delete-namespace-on-failure=false
157+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:qat.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
151158

152159
e2e-sgx:
153-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "SGX" -delete-namespace-on-failure=false
160+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:sgx.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
154161

155162
e2e-gpu:
156-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "GPU" -delete-namespace-on-failure=false
163+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:gpu.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
157164

158165
e2e-dsa:
159-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "DSA" -delete-namespace-on-failure=false
166+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:dsa.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
160167

161168
e2e-iaa:
162-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "IAA" -delete-namespace-on-failure=false
169+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:iaa.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
163170

164171
e2e-dlb:
165-
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "DLB" -delete-namespace-on-failure=false
172+
@$(GO) test -v ./test/e2e/... -ginkgo.v -ginkgo.show-node-events -ginkgo.focus "Device:dlb.*$(ADDITIONAL_FOCUS_REGEX)" $(GENERATED_SKIP_OPT) -delete-namespace-on-failure=false
166173

167174
terrascan:
168175
@ls deployments/*/kustomization.yaml | while read f ; \

‎test/e2e/dlb/dlb.go

+13-5
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ const (
3939
)
4040

4141
func init() {
42-
ginkgo.Describe("DLB plugin", describe)
42+
ginkgo.Describe("DLB plugin [Device:dlb]", describe)
4343
}
4444

4545
func describe() {
@@ -81,30 +81,38 @@ func describe() {
8181
}
8282
})
8383

84-
ginkgo.Context("When PF resources are available", func() {
84+
ginkgo.Context("When PF resources are available [Resource:pf]", func() {
8585
ginkgo.BeforeEach(func(ctx context.Context) {
8686
resource := v1.ResourceName("dlb.intel.com/pf")
8787
if err := utils.WaitForNodesWithResource(ctx, f.ClientSet, resource, 30*time.Second); err != nil {
8888
framework.Failf("unable to wait for nodes to have positive allocatable resource %s: %v", resource, err)
8989
}
9090
})
9191

92-
ginkgo.It("can run demo app", func(ctx context.Context) {
92+
ginkgo.It("can run demo app [App:libdlb]", func(ctx context.Context) {
9393
runDemoApp(ctx, "PF", demoPFYaml, f)
9494
})
95+
96+
ginkgo.When("there is no app to run [App:noapp]", func() {
97+
ginkgo.It("does nothing", func() {})
98+
})
9599
})
96100

97-
ginkgo.Context("When VF resources are available", func() {
101+
ginkgo.Context("When VF resources are available [Resource:vf]", func() {
98102
ginkgo.BeforeEach(func(ctx context.Context) {
99103
resource := v1.ResourceName("dlb.intel.com/vf")
100104
if err := utils.WaitForNodesWithResource(ctx, f.ClientSet, resource, 30*time.Second); err != nil {
101105
framework.Failf("unable to wait for nodes to have positive allocatable resource %s: %v", resource, err)
102106
}
103107
})
104108

105-
ginkgo.It("can run demo app", func(ctx context.Context) {
109+
ginkgo.It("can run demo app [App:libdlb]", func(ctx context.Context) {
106110
runDemoApp(ctx, "VF", demoVFYaml, f)
107111
})
112+
113+
ginkgo.When("there is no app to run [App:noapp]", func() {
114+
ginkgo.It("does nothing", func() {})
115+
})
108116
})
109117
}
110118

‎test/e2e/dsa/dsa.go

+7-3
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ const (
4040
)
4141

4242
func init() {
43-
ginkgo.Describe("DSA plugin", describe)
43+
ginkgo.Describe("DSA plugin [Device:dsa]", describe)
4444
}
4545

4646
func describe() {
@@ -94,20 +94,24 @@ func describe() {
9494
}
9595
})
9696

97-
ginkgo.Context("When DSA resources are available", func() {
97+
ginkgo.Context("When DSA resources are available [Resource:dedicated]", func() {
9898
ginkgo.BeforeEach(func(ctx context.Context) {
9999
ginkgo.By("checking if the resource is allocatable")
100100
if err := utils.WaitForNodesWithResource(ctx, f.ClientSet, "dsa.intel.com/wq-user-dedicated", 300*time.Second); err != nil {
101101
framework.Failf("unable to wait for nodes to have positive allocatable resource: %v", err)
102102
}
103103
})
104104

105-
ginkgo.It("deploys a demo app", func(ctx context.Context) {
105+
ginkgo.It("deploys a demo app [App:accel-config]", func(ctx context.Context) {
106106
e2ekubectl.RunKubectlOrDie(f.Namespace.Name, "apply", "-f", demoPath)
107107

108108
ginkgo.By("waiting for the DSA demo to succeed")
109109
err := e2epod.WaitForPodSuccessInNamespaceTimeout(ctx, f.ClientSet, podName, f.Namespace.Name, 200*time.Second)
110110
gomega.Expect(err).To(gomega.BeNil(), utils.GetPodLogs(ctx, f, podName, podName))
111111
})
112+
113+
ginkgo.When("there is no app to run [App:noapp]", func() {
114+
ginkgo.It("does nothing", func() {})
115+
})
112116
})
113117
}

‎test/e2e/fpga/fpga.go

+25-5
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ const (
4747
)
4848

4949
func init() {
50-
ginkgo.Describe("FPGA Plugin", describe)
50+
ginkgo.Describe("FPGA Plugin [Device:fpga]", describe)
5151
}
5252

5353
func describe() {
@@ -64,23 +64,43 @@ func describe() {
6464
fmw := framework.NewDefaultFramework("fpgaplugin-e2e")
6565
fmw.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged
6666

67-
ginkgo.Context("When FPGA plugin is running in region mode", func() {
67+
ginkgo.Context("When FPGA plugin is running in region mode [Mode:region]", func() {
6868
ginkgo.BeforeEach(func(ctx context.Context) {
6969
runDevicePlugin(ctx, fmw, pluginKustomizationPath, mappingsCollectionPath, arria10NodeResource, "region")
7070
})
71-
ginkgo.It("runs an opae-nlb-demo pod two times", func(ctx context.Context) {
71+
ginkgo.It("runs an opae-nlb-demo pod two times [App:opae-nlb-demo]", func(ctx context.Context) {
7272
runTestCase(ctx, fmw, "region", nlb3PodResource, "nlb3", "nlb0")
7373
runTestCase(ctx, fmw, "region", nlb0PodResource, "nlb0", "nlb3")
7474
})
7575
})
7676

77-
ginkgo.Context("When FPGA plugin is running in af mode", func() {
77+
ginkgo.Context("When FPGA plugin is running in af mode [Mode:af]", func() {
7878
ginkgo.BeforeEach(func(ctx context.Context) {
7979
runDevicePlugin(ctx, fmw, pluginKustomizationPath, mappingsCollectionPath, nlb0NodeResource, "af")
8080
})
81-
ginkgo.It("runs an opae-nlb-demo pod", func(ctx context.Context) {
81+
ginkgo.It("runs an opae-nlb-demo pod [App:opae-nlb-demo]", func(ctx context.Context) {
8282
runTestCase(ctx, fmw, "af", nlb0PodResourceAF, "nlb0", "nlb3")
8383
})
84+
85+
ginkgo.When("there is no app to run [App:noapp]", func() {
86+
ginkgo.It("does nothing", func() {})
87+
})
88+
})
89+
90+
ginkgo.Context("When FPGA plugin is running in region mode [Mode:region]", func() {
91+
ginkgo.BeforeEach(func(ctx context.Context) {
92+
runDevicePlugin(ctx, fmw, pluginKustomizationPath, mappingsCollectionPath, arria10NodeResource, "region")
93+
})
94+
ginkgo.It("runs [App:opae-nlb-demo]", func(ctx context.Context) {
95+
runTestCase(ctx, fmw, "region", nlb3PodResource, "nlb3", "nlb0")
96+
})
97+
ginkgo.It("runs an opae-nlb-demo pod [App:opae-nlb-demo]", func(ctx context.Context) {
98+
runTestCase(ctx, fmw, "region", nlb0PodResource, "nlb0", "nlb3")
99+
})
100+
101+
ginkgo.When("there is no app to run [App:noapp]", func() {
102+
ginkgo.It("does nothing", func() {})
103+
})
84104
})
85105
}
86106

‎test/e2e/gpu/gpu.go

+8-4
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ const (
4343
)
4444

4545
func init() {
46-
ginkgo.Describe("GPU plugin", describe)
46+
ginkgo.Describe("GPU plugin [Device:gpu]", describe)
4747
}
4848

4949
func describe() {
@@ -74,14 +74,14 @@ func describe() {
7474
}
7575
})
7676

77-
ginkgo.Context("When GPU resources are available", func() {
77+
ginkgo.Context("When GPU resources are available [Resource:i915]", func() {
7878
ginkgo.BeforeEach(func(ctx context.Context) {
7979
ginkgo.By("checking if the resource is allocatable")
8080
if err := utils.WaitForNodesWithResource(ctx, f.ClientSet, "gpu.intel.com/i915", 30*time.Second); err != nil {
8181
framework.Failf("unable to wait for nodes to have positive allocatable resource: %v", err)
8282
}
8383
})
84-
ginkgo.It("checks availability of GPU resources", func(ctx context.Context) {
84+
ginkgo.It("checks availability of GPU resources [App:busybox]", func(ctx context.Context) {
8585
ginkgo.By("submitting a pod requesting GPU resources")
8686
podSpec := &v1.Pod{
8787
ObjectMeta: metav1.ObjectMeta{Name: "gpuplugin-tester"},
@@ -122,7 +122,7 @@ func describe() {
122122
framework.Logf("found card and renderD from the log")
123123
})
124124

125-
ginkgo.It("run a small workload on the GPU", func(ctx context.Context) {
125+
ginkgo.It("run a small workload on the GPU [App:tensorflow]", func(ctx context.Context) {
126126
kustomYaml, err := utils.LocateRepoFile(tfKustomizationYaml)
127127
if err != nil {
128128
framework.Failf("unable to locate %q: %v", tfKustomizationYaml, err)
@@ -139,5 +139,9 @@ func describe() {
139139

140140
framework.Logf("tensorflow execution succeeded!")
141141
})
142+
143+
ginkgo.When("there is no app to run [App:noapp]", func() {
144+
ginkgo.It("does nothing", func() {})
145+
})
142146
})
143147
}

0 commit comments

Comments
 (0)