anchor.diff:  # Thread 8: 45766 56169 56171 ...# 00015 others
   # Thread 9: 02974 08094 13214 ... # 00090 others
   ----
  -<1> The docker arguments provided by the user.
  -<2> The user is notified about the measured latency and the configured latency.
  +<1> The `podman` arguments you provided.
  +<2> You can see the measured latency and the configured latency.
   <3> The arguments for the `cyclictest` command.
   <4> The maximum latencies measured on each thread.
   
  +[id="cnf-performing-end-to-end-tests-capturing-results-cyclictest"]
  +==== Capturing the results
  +
  +The same output can indicate different results for different workloads. For example, spikes up to 18μs is acceptable for 4G DU workloads but not for 5G DU workloads. Spikes above 20μs are not acceptable in any case.
  +
  +.Example of good results
  +
  +[source, terminal]
  +----
  +running cmd: cyclictest -q -D 10m -p 1 -t 16 -a 2,4,6,8,10,12,14,16,54,56,58,60,62,64,66,68 -h 30 -i 1000 -m 
  +# Histogram
  +000000 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  +000001 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  +000002 579506	535967	418614	573648	532870	529897	489306	558076	582350	585188	583793	223781	532480	569130	472250	576043
  +More histogram entries ...
  +# Total: 000600000 000600000 000600000 000599999 000599999 000599999 000599998 000599998 000599998 000599997 000599997 000599996 000599996 000599995 000599995 000599995
  +# Min Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
  +# Avg Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
  +# Max Latencies: 00005 00005 00004 00005 00004 00004 00005 00005 00006 00005 00004 00005 00004 00004 00005 00004
  +# Histogram Overflows: 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000
  +# Histogram Overflow at cycle number:
  +# Thread 0:
  +# Thread 1:
  +# Thread 2:
  +# Thread 3:
  +# Thread 4:
  +# Thread 5:
  +# Thread 6:
  +# Thread 7:
  +# Thread 8:
  +# Thread 9:
  +# Thread 10:
  +# Thread 11:
  +# Thread 12:
  +# Thread 13:
  +# Thread 14:
  +# Thread 15:
  +----
  +
  +.Example of bad results
  +
  +[source, terminal]
  +----
  +running cmd: cyclictest -q -D 10m -p 1 -t 16 -a 2,4,6,8,10,12,14,16,54,56,58,60,62,64,66,68 -h 30 -i 1000 -m 
  +# Histogram
  +000000 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  +000001 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  +000002 564632	579686	354911	563036	492543	521983	515884	378266	592621	463547	482764	591976	590409	588145	589556	353518
  +More histogram entries ...
  +# Total: 000599999 000599999 000599999 000599997 000599997 000599998 000599998 000599997 000599997 000599996 000599995 000599996 000599995 000599995 000599995 000599993
  +# Min Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
  +# Avg Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
  +# Max Latencies: 00493 00387 00271 00619 00541 00513 00009 00389 00252 00215 00539 00498 00363 00204 00068 00520
  +# Histogram Overflows: 00001 00001 00001 00002 00002 00001 00000 00001 00001 00001 00002 00001 00001 00001 00001 00002
  +# Histogram Overflow at cycle number:
  +# Thread 0: 155922
  +# Thread 1: 110064
  +# Thread 2: 110064
  +# Thread 3: 110063 155921
  +# Thread 4: 110063 155921
  +# Thread 5: 155920
  +# Thread 6:
  +# Thread 7: 110062
  +# Thread 8: 110062
  +# Thread 9: 155919
  +# Thread 10: 110061 155919
  +# Thread 11: 155918
  +# Thread 12: 155918
  +# Thread 13: 110060
  +# Thread 14: 110060
  +# Thread 15: 110059 155917
  +----
  +
   [id="cnf-performing-end-to-end-tests-running-oslat"]
   === Running oslat
   
  -[source, terminal]
  -----
  -$ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_CPUS=7 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.10 /usr/bin/test-run.sh -ginkgo.focus="oslat"
  -----
  -
  -The above command runs the `cyclictest` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  +.Prerequisites
  +
  +* You logged into registry.redhat.io with your Customer Portal credentials
  +* You installed the Performance Add-on Operator and you applied a performance profile
  +
  +.Procedure
  +
  +* To perform the `oslat`, run the following command: 
  +
  +[source,terminal,subs="attributes+"]
  +----
  +$ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e \
  +LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e ROLE_WORKER_CNF=worker-cnf -e \
  +LATENCY_TEST_CPUS=7 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
  +registry.redhat.io/openshift4/cnf-tests-rhel8:v{product-version} /usr/bin/test-run.sh -ginkgo.focus="oslat"
  +----
  +
  +The command runs the `oslat` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!` when this test is completed.
  +
  +If the results exceed the latency threshold, the test fails and you can see the following output:
   
   .Example failure output
  -[source, terminal]
  -----
  -$ docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e IMAGE_REGISTRY=quay.io/titzhak -e CNF_TESTS_IMAGE=cnf-tests:latest -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e DISCOVERY_MODE=true -e MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=7 quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="oslat" <1>
  +[source,terminal,subs="attributes+"]
  +----
  +$ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e \
  +IMAGE_REGISTRY="registry.redhat.io/openshift4/" -e CNF_TESTS_IMAGE=cnf-tests-rhel8:v{product-version} -e \
  +PERF_TEST_PROFILE=<performance_profile_name> -e ROLE_WORKER_CNF=worker-cnf -e \
  +LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e DISCOVERY_MODE=true -e \
  +MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=7 \
  +registry.redhat.io/openshift4/cnf-tests-rhel8:v{product-version} \
  +usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="oslat" <1>
   
   running /usr/bin//validationsuite -ginkgo.v -ginkgo.focus=oslat
   I0829 12:36:55.386776       8 request.go:668] Waited for 1.000303471s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/authentication.k8s.io/v1?timeout=32s
  
positive.code: Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  I0811 15:02:51.092254       1 node.go:37] Environment information: /proc/cmdline: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/vmlinuz-4.18.0-305.10.2.rt7.83.el8_4.x86_64 ip=dhcp random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/rhcos/612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/0 ignition.platform.id=openstack root=UUID=5a4ddf16-9372-44d9-ac4e-3ee329e16ab3 rw rootflags=prjquota skew_tick=1 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=000000ff,ffffffff,ffffffff,fffffff1 intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt isolcpus=managed_irq,1-3 systemd.cpu_affinity=0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 default_hugepagesz=1G hugepagesz=2M hugepages=128 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0
  I0811 15:02:51.092427       1 node.go:44] Environment information: kernel version 4.18.0-305.10.2.rt7.83.el8_4.x86_64
  I0811 15:02:51.092450       1 main.go:48] running the cyclictest command with arguments [-D 600 -p 1 -t 10 -a 2,4,6,8,10,54,56,58,60,62 -h 30 -i 1000 --quiet] <3>
  I0811 15:03:06.147253       1 main.go:54] succeeded to run the cyclictest command: # /dev/cpu_dma_latency set to 0us
  # Histogram
  000000 000000	000000	000000	000000	000000	000000	000000	000000	000000	000000
  
negative.code: 
  [id="cnf-performing-end-to-end-tests-running-the-tests_{context}"]
  == Running the latency tests
  
  If the `kubeconfig` file is in the current folder, you can run the test suite by using the following command:
  
  [source,terminal]
  ----
  $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e \
  DISCOVERY_MODE=true registry.redhat.io/openshift4/cnf-tests-rhel8:v{product-version} \
  /usr/bin/test-run.sh -ginkgo.focus="\[performance\]\ Latency\ Test"
  ----
  
  This allows the running container to use the `kubeconfig` file from inside the container.
  
  [WARNING]
  ====
  You must run the latency tests in Discovery mode. The latency tests can change the configuration of your cluster if you do not run in Discovery mode.
  ====
  
  In {product-title} {product-version}, you can also run latency tests from the CNF-test container. The latency test allows you to validate node tuning for your workload.
  
  Three tools measure the latency of the system:
  
  