anchor.diff:  [id="cnf-performing-end-to-end-tests-cleaning-up_{context}"]
   === Cleaning up
   
  -After running the test suite, all the dangling resources are cleaned up.
  +After running the test suite, all the dangling resources are cleaned up.
positive.code: ====
  
  [id="cnf-performing-end-to-end-tests-running-the-tests_{context}"]
  == Running the tests
  Assuming the `kubeconfig` file is in the current folder, the command for running the test suite is:
  
  [source,terminal]
  
negative.code: 
  This allows your `kubeconfig` file to be consumed from inside the running container.
  
  [WARNING]
  ====
  You must run the latency tests in Discovery mode. The latency tests can change the configuration of your cluster if you do not run in Discovery mode.
  ====
  
  In {product-title} {product-version}, you can also run latency tests from the CNF-test container. The latency test allows the to validate that node tuning is sufficient for your workload.
  
  Three tools measure the latency of the system:
  
  * `hwlatdetect`
  * `cyclictest`
  * `oslat`
  
  Each tool has a specific use. Use the tools in sequence to achieve reliable test results.
  
  . The `hwlatdetect` tool measures the baseline that the bare hardware can achieve. Before proceeding with the next latency test, ensure that the number measured by `hwlatdetect` meets the required threshold because hardware latency spikes cannot be fixed by operating system tuning.
  
  .  The `cyclictest` tool verifies the timer latency after `hwlatdetect` passes validation. The `cyclictest` tool schedules a repeated timer and measures the difference between the desired and the actual trigger times. The difference can uncover basic issues with the tuning caused by interrupts or process priorities.
  
  . The `oslat` tool behaves similarly to a CPU-intensive DPDK application and measures all the interruptions and disruptions to the busy loop that simulates CPU heavy data processing.
  
  By default, the latency tests are disabled. To enable the latency test, you must add the `LATENCY_TEST_RUN` environment variable to the test invocation and set its value to `true`. For example, `LATENCY_TEST_RUN=true`.
  
  The test introduces the following environment variables:
  
  * `LATENCY_TEST_CPUS` variable specifies the number of CPUs that the pod running the latency tests uses.
  * `LATENCY_TEST_RUNTIME` variable specifies the amount of time in seconds that the latency test must run.
  * `CYCLICTEST_MAXIMUM_LATENCY` variable specifies the maximum latency in microseconds that all threads expect before waking up during the `cyclictest` run.
  * `HWLATDETECT_MAXIMUM_LATENCY` variable specifies the maximum acceptable hardware latency in microseconds for the workload and operating system.
  * `OSLAT_MAXIMUM_LATENCY` variable specifies the maximum acceptable latency in microseconds for the `oslat` test results.
  * `MAXIMUM_LATENCY` is a unified variable you can apply for all tests.
  
  [NOTE]
  ====
  A variable that is specific to certain tests has precedence over the unified variable.
  ====
  
  You can use the `ginkgo.focus` flag to run a specific test.
  
  [id="cnf-performing-end-to-end-tests-running-hwlatdetect"]
  === Running hwlatdetect
  
  To perform the `hwlatdetect`, run the following command:
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DICOVERY_MODE=true -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.9 /usr/bin/test-run.sh -ginko.focus=”hwladetect”
  ----
  
  The above command runs the `hwlatdetect` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $ docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=40 -e MAXIMUM_LATENCY=1 -e LATENCY_TEST_CPUS=10 -e DISCOVERY_MODE=true quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.focus="hwlatdetect" <1>
  running /usr/bin//validationsuite -ginkgo.focus=hwlatdetect
  ...
  Discovery mode enabled, skipping setup
  running /usr/bin//cnftests -ginkgo.focus=hwlatdetect
  I0812 09:53:57.108148      19 request.go:668] Waited for 1.049207747s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/autoscaling/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1628762033
  Will run 1 of 138 specs
  
  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
  ------------------------------
  • [SLOW TEST:26.144 seconds]
  [performance] Latency Test
  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:84
    with the hwlatdetect image
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:224
      should succeed
      /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:232
  ------------------------------
  SSSSSSSSSSSSSSSSSSSSSS
  
  detector: tracer
     parameters:
          Latency threshold: 1us <2>
          Sample window:     10000000us
          Sample width:      950000us
       Non-sampling period:  9050000us
          Output File:       None
  
  Starting test
  test finished
  Max Latency: 35us <3>
  Samples recorded: 2
  Samples exceeding threshold: 2
  ts: 1628174377.074638224, inner:20, outer:35
  ts: 1628174387.359881340, inner:21, outer:34
  ; err: exit status 1
  goroutine 1 [running]:
  k8s.io/klog.stacks(0xc000070200, 0xc000106400, 0x21b, 0x3c2)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:875 +0xb9
  k8s.io/klog.(*loggingT).output(0x5bed00, 0xc000000003, 0xc00010a0e0, 0x53ea81, 0x7, 0x33, 0x0)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:826 +0x35f
  k8s.io/klog.(*loggingT).printf(0x5bed00, 0x3, 0x5082da, 0x33, 0xc000113f58, 0x2, 0x2)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:707 +0x153
  k8s.io/klog.Fatalf(...)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:1276
  main.main()
  	/remote-source/app/cnf-tests/pod-utils/hwlatdetect-runner/main.go:51 +0x897
  ----
  <1> The docker arguments provided by the user.
  <2> The latency threshold configured by the user using the `MAX_LATENCY` or the `HWLATDETECT_MAX_LATENCY` environment variables.
  <3> The maximum latency value measured during the test.
  
  [id="cnf-performing-end-to-end-tests-running-cyclictest"]
  === Running cyclictest
  
  To perform the `cyclictest`, run the following command:
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DICOVERY_MODE=true -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.9 /usr/bin/test-run.sh -ginkgo.focus="cyclictest"
  ----
  
  The above command runs the `cyclictest` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=10 -e DISCOVERY_MODE=true quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="cyclictest" <1>
  
  Discovery mode enabled, skipping setup
  running /usr/bin//cnftests -ginkgo.v -ginkgo.focus=cyclictest
  I0811 15:02:36.350033      20 request.go:668] Waited for 1.049965918s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/machineconfiguration.openshift.io/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1628694153
  Will run 1 of 138 specs
  
  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
  ------------------------------
  [performance] Latency Test with the cyclictest image
    should succeed
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:200
  STEP: Waiting two minutes to download the latencyTest image
  STEP: Waiting another two minutes to give enough time for the cluster to move the pod to Succeeded phase
  Aug 11 15:03:06.826: [INFO]: found mcd machine-config-daemon-wf4w8 for node cnfdc8.clus2.t5g.lab.eng.bos.redhat.com
  
  • Failure [22.527 seconds]
  [performance] Latency Test
  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:84
    with the cyclictest image
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:188
      should succeed [It]
      /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:200
  
      The current latency 17 is bigger than the expected one 20 <2>
      Expected
          <bool>: false
      to be true
  
      /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:219
  
  Log file created at: 2021/08/11 15:02:51
  Running on machine: cyclictest-knk7d
  Binary: Built with gc go1.16.6 for linux/amd64
  Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  I0811 15:02:51.092254       1 node.go:37] Environment information: /proc/cmdline: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/vmlinuz-4.18.0-305.10.2.rt7.83.el8_4.x86_64 ip=dhcp random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/rhcos/612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/0 ignition.platform.id=openstack root=UUID=5a4ddf16-9372-44d9-ac4e-3ee329e16ab3 rw rootflags=prjquota skew_tick=1 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=000000ff,ffffffff,ffffffff,fffffff1 intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt isolcpus=managed_irq,1-3 systemd.cpu_affinity=0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 default_hugepagesz=1G hugepagesz=2M hugepages=128 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0
  I0811 15:02:51.092427       1 node.go:44] Environment information: kernel version 4.18.0-305.10.2.rt7.83.el8_4.x86_64
  I0811 15:02:51.092450       1 main.go:48] running the cyclictest command with arguments [-D 600 -p 1 -t 10 -a 2,4,6,8,10,54,56,58,60,62 -h 30 -i 1000 --quiet] <3>
  I0811 15:03:06.147253       1 main.go:54] succeeded to run the cyclictest command: # /dev/cpu_dma_latency set to 0us
  # Histogram
  000000 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  000001 000000	005561	027778	037704	011987	000000	120755	238981	081847	300186
  000002 587440	581106	564207	554323	577416	590635	474442	357940	513895	296033
  000003 011751	011441	006449	006761	008409	007904	002893	002066	003349	003089
  000004 000527	001079	000914	000712	001451	001120	000779	000283	000350	000251
  
  More histogram entries ...
  # Min Latencies: 00002 00001 00001 00001 00001 00002 00001 00001 00001 00001
  # Avg Latencies: 00002 00002 00002 00001 00002 00002 00001 00001 00001 00001
  # Max Latencies: 00018 00465 00361 00395 00208 00301 02052 00289 00327 00114 <4>
  # Histogram Overflows: 00000 00220 00159 00128 00202 00017 00069 00059 00045 00120
  # Histogram Overflow at cycle number:
  # Thread 0:
  # Thread 1: 01142 01439 05305 … # 00190 others
  # Thread 2: 20895 21351 30624 … # 00129 others
  # Thread 3: 01143 17921 18334 … # 00098 others
  # Thread 4: 30499 30622 31566 ... # 00172 others
  # Thread 5: 145221 170910 171888 ...
  # Thread 6: 01684 26291 30623 ...# 00039 others
  # Thread 7: 28983 92112 167011 … 00029 others
  # Thread 8: 45766 56169 56171 ...# 00015 others
  # Thread 9: 02974 08094 13214 ... # 00090 others
  ----
  <1> The docker arguments provided by the user.
  <2> The user is notified about the measured latency and the configured latency.
  <3> The arguments for the `cyclictest` command.
  <4> The maximum latencies measured on each thread.
  
  [id="cnf-performing-end-to-end-tests-running-oslat"]
  === Running oslat
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DICOVERY_MODE=true -e LATENCY_TEST_CPUS=7 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.9 /usr/bin/test-run.sh -ginkgo.focus="oslat"
  ----
  
  The above command runs the `cyclictest` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $ docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e IMAGE_REGISTRY=quay.io/titzhak -e CNF_TESTS_IMAGE=cnf-tests:latest -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e DISCOVERY_MODE=true -e MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=7 quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="oslat" <1>
  
  running /usr/bin//validationsuite -ginkgo.v -ginkgo.focus=oslat
  I0829 12:36:55.386776       8 request.go:668] Waited for 1.000303471s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/authentication.k8s.io/v1?timeout=32s
  Running Suite: CNF Features e2e validation
  ==========================================
  
  Discovery mode enabled, skipping setup
  running /usr/bin//cnftests -ginkgo.v -ginkgo.focus=oslat
  I0829 12:37:01.219077      20 request.go:668] Waited for 1.050010755s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/snapshot.storage.k8s.io/v1beta1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1630240617
  Will run 1 of 142 specs
  
  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
  ------------------------------
  [performance] Latency Test with the oslat image
    should succeed
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:134
  STEP: Waiting two minutes to download the latencyTest image
  STEP: Waiting another two minutes to give enough time for the cluster to move the pod to Succeeded phase
  Aug 29 12:37:59.324: [INFO]: found mcd machine-config-daemon-wf4w8 for node cnfdc8.clus2.t5g.lab.eng.bos.redhat.com
  
  • Failure [49.246 seconds]
  [performance] Latency Test
  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:59
    with the oslat image
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:112
      should succeed [It]
      /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:134
  
      The current latency 27 is bigger than the expected one 20 <2>
      Expected
          <bool>: false
      to be true
   /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:168
  
  Log file created at: 2021/08/29 13:25:21
  Running on machine: oslat-57c2g
  Binary: Built with gc go1.16.6 for linux/amd64
  Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  I0829 13:25:21.569182       1 node.go:37] Environment information: /proc/cmdline: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/vmlinuz-4.18.0-305.10.2.rt7.83.el8_4.x86_64 ip=dhcp random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/rhcos/612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/0 ignition.platform.id=openstack root=UUID=5a4ddf16-9372-44d9-ac4e-3ee329e16ab3 rw rootflags=prjquota skew_tick=1 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=000000ff,ffffffff,ffffffff,fffffff1 intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt isolcpus=managed_irq,1-3 systemd.cpu_affinity=0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 default_hugepagesz=1G hugepagesz=2M hugepages=128 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0
  I0829 13:25:21.569345       1 node.go:44] Environment information: kernel version 4.18.0-305.10.2.rt7.83.el8_4.x86_64
  I0829 13:25:21.569367       1 main.go:53] Running the oslat command with arguments [--duration 600 --rtprio 1 --cpu-list 4,6,52,54,56,58 --cpu-main-thread 2] <1>
  I0829 13:35:22.632263       1 main.go:59] Succeeded to run the oslat command: oslat V 2.00
  Total runtime: 		600 seconds
  Thread priority: 	SCHED_FIFO:1
  CPU list: 		4,6,52,54,56,58
  CPU for main thread: 	2
  Workload: 		no
  Workload mem: 		0 (KiB)
  Preheat cores: 		6
  
  Pre-heat for 1 seconds...
  Test starts...
  Test completed.
  
          Core:	 4 6 52 54 56 58
      CPU Freq:	 2096 2096 2096 2096 2096 2096 (Mhz)
      001 (us):	 19390720316 19141129810 20265099129 20280959461 19391991159 19119877333
      002 (us):	 5304 5249 5777 5947 6829 4971
      003 (us):	 28 14 434 47 208 21
      004 (us):	 1388 853 123568 152817 5576 0
      005 (us):	 207850 223544 103827 91812 227236 231563
      006 (us):	 60770 122038 277581 323120 122633 122357
      007 (us):	 280023 223992 63016 25896 214194 218395
      008 (us):	 40604 25152 24368 4264 24440 25115
      009 (us):	 6858 3065 5815 810 3286 2116
      010 (us):	 1947 936 1452 151 474 361
    ...
       Minimum:	 1 1 1 1 1 1 (us)
       Average:	 1.000 1.000 1.000 1.000 1.000 1.000 (us)
       Maximum:	 37 38 49 28 28 19 (us) <3>
       Max-Min:	 36 37 48 27 27 18 (us)
      Duration:	 599.667 599.667 599.667 599.667 599.667 599.667 (sec)
  ----
  <1> The list of CPUs running the `oslat` command. Seven CPUs are provided through the `LATENCY_TEST_CPUS` variable. Only six CPUs are displayed in total because one is used to run the `oslat` tool.
  <2> The user is notified about the measured latency and the configured latency.
  <3> The maximum latency values in microseconds that are measured on each CPU.
  
  [id="cnf-performing-end-to-end-tests-image-parameters_{context}"]
  == Image parameters
  