anchor.diff:  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e CNF_TESTS_IMAGE="custom-cnf-tests-image:latests" registry.redhat.io/openshift4/cnf-tests-rhel8:v4.10 /usr/bin/test-run.sh
   ----
   
  -[id="cnf-performing-end-to-end-tests-ginko-parameters_{context}"]
  +[id="cnf-performing-end-to-end-tests-ginkgo-parameters_{context}"]
   === Ginkgo parameters
   
  -The test suite is built upon the ginkgo BDD framework. This means that it accepts parameters for filtering or skipping tests.
  +The Ginkgo BDD (Behavior-Driven Development) framework serves as the base for the test suite. This means that it accepts parameters for filtering or skipping tests.
   
   You can use the `-ginkgo.focus` parameter to filter a set of tests:
   
  
positive.code: 
  The test introduces the following environment variables:
  
  * `LATENCY_TEST_CPUS` variable specifies the number of CPUs that the pod running the latency tests uses.
  * `LATENCY_TEST_RUNTIME` variable specifies the amount of time in seconds that the latency test must run.
  * `CYCLICTEST_MAXIMUM_LATENCY` variable specifies the maximum latency in microseconds that all threads expect before waking up during the `cyclictest` run.
  * `HWLATDETECT_MAXIMUM_LATENCY` variable specifies the maximum acceptable hardware latency in microseconds for the workload and operating system.
  * `OSLAT_MAXIMUM_LATENCY` variable specifies the maximum acceptable latency in microseconds for the `oslat` test results.
  * `MAXIMUM_LATENCY` is a unified variable you can apply for all tests.
  
  [NOTE]
  ====
  A variable that is specific to certain tests has precedence over the unified variable.
  ====
  
  You can use the `ginkgo.focus` flag to run a specific test.
  
  [id="cnf-performing-end-to-end-tests-running-hwlatdetect"]
  === Running hwlatdetect
  
  To perform the `hwlatdetect`, run the following command:
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.10 /usr/bin/test-run.sh -ginko.focus=”hwladetect”
  ----
  
  The above command runs the `hwlatdetect` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $ docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=40 -e MAXIMUM_LATENCY=1 -e LATENCY_TEST_CPUS=10 -e DISCOVERY_MODE=true quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.focus="hwlatdetect" <1>
  running /usr/bin//validationsuite -ginkgo.focus=hwlatdetect
  ...
  Discovery mode enabled, skipping setup
  running /usr/bin//cnftests -ginkgo.focus=hwlatdetect
  I0812 09:53:57.108148      19 request.go:668] Waited for 1.049207747s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/autoscaling/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1628762033
  Will run 1 of 138 specs
  
  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
  ------------------------------
  • [SLOW TEST:26.144 seconds]
  [performance] Latency Test
  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:84
    with the hwlatdetect image
    /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:224
      should succeed
      /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift-kni/performance-addon-operators/functests/4_latency/latency.go:232
  ------------------------------
  SSSSSSSSSSSSSSSSSSSSSS
  
  detector: tracer
     parameters:
          Latency threshold: 1us <2>
          Sample window:     10000000us
          Sample width:      950000us
       Non-sampling period:  9050000us
          Output File:       None
  
  Starting test
  test finished
  Max Latency: 35us <3>
  Samples recorded: 2
  Samples exceeding threshold: 2
  ts: 1628174377.074638224, inner:20, outer:35
  ts: 1628174387.359881340, inner:21, outer:34
  ; err: exit status 1
  goroutine 1 [running]:
  k8s.io/klog.stacks(0xc000070200, 0xc000106400, 0x21b, 0x3c2)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:875 +0xb9
  k8s.io/klog.(*loggingT).output(0x5bed00, 0xc000000003, 0xc00010a0e0, 0x53ea81, 0x7, 0x33, 0x0)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:826 +0x35f
  k8s.io/klog.(*loggingT).printf(0x5bed00, 0x3, 0x5082da, 0x33, 0xc000113f58, 0x2, 0x2)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:707 +0x153
  k8s.io/klog.Fatalf(...)
  	/remote-source/app/vendor/k8s.io/klog/klog.go:1276
  main.main()
  	/remote-source/app/cnf-tests/pod-utils/hwlatdetect-runner/main.go:51 +0x897
  ----
  <1> The docker arguments provided by the user.
  <2> The latency threshold configured by the user using the `MAX_LATENCY` or the `HWLATDETECT_MAX_LATENCY` environment variables.
  <3> The maximum latency value measured during the test.
  
  [id="cnf-performing-end-to-end-tests-running-cyclictest"]
  === Running cyclictest
  
  To perform the `cyclictest`, run the following command:
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.10 /usr/bin/test-run.sh -ginkgo.focus="cyclictest"
  ----
  
  The above command runs the `cyclictest` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=10 -e DISCOVERY_MODE=true quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="cyclictest" <1>
  
  Discovery mode enabled, skipping setup
  running /usr/bin//cnftests -ginkgo.v -ginkgo.focus=cyclictest
  
negative.code: Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  I0811 15:02:51.092254       1 node.go:37] Environment information: /proc/cmdline: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/vmlinuz-4.18.0-305.10.2.rt7.83.el8_4.x86_64 ip=dhcp random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/rhcos/612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/0 ignition.platform.id=openstack root=UUID=5a4ddf16-9372-44d9-ac4e-3ee329e16ab3 rw rootflags=prjquota skew_tick=1 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=000000ff,ffffffff,ffffffff,fffffff1 intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt isolcpus=managed_irq,1-3 systemd.cpu_affinity=0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 default_hugepagesz=1G hugepagesz=2M hugepages=128 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0
  I0811 15:02:51.092427       1 node.go:44] Environment information: kernel version 4.18.0-305.10.2.rt7.83.el8_4.x86_64
  I0811 15:02:51.092450       1 main.go:48] running the cyclictest command with arguments \
  [-D 600 -95 1 -t 10 -a 2,4,6,8,10,54,56,58,60,62 -h 30 -i 1000 --quiet] <3>
  I0811 15:03:06.147253       1 main.go:54] succeeded to run the cyclictest command: # /dev/cpu_dma_latency set to 0us
  # Histogram
  000000 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  