anchor.diff:  Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
   I0829 13:25:21.569182       1 node.go:37] Environment information: /proc/cmdline: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/vmlinuz-4.18.0-305.10.2.rt7.83.el8_4.x86_64 ip=dhcp random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/rhcos/612d89f4519a53ad0b1a132f4add78372661bfb3994f5fe115654971aa58a543/0 ignition.platform.id=openstack root=UUID=5a4ddf16-9372-44d9-ac4e-3ee329e16ab3 rw rootflags=prjquota skew_tick=1 nohz=on rcu_nocbs=1-3 tuned.non_isolcpus=000000ff,ffffffff,ffffffff,fffffff1 intel_pstate=disable nosoftlockup tsc=nowatchdog intel_iommu=on iommu=pt isolcpus=managed_irq,1-3 systemd.cpu_affinity=0,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 default_hugepagesz=1G hugepagesz=2M hugepages=128 nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0
   I0829 13:25:21.569345       1 node.go:44] Environment information: kernel version 4.18.0-305.10.2.rt7.83.el8_4.x86_64
  -I0829 13:25:21.569367       1 main.go:53] Running the oslat command with arguments [--duration 600 --rtprio 1 --cpu-list 4,6,52,54,56,58 --cpu-main-thread 2] <1>
  +I0829 13:25:21.569367       1 main.go:53] Running the oslat command with arguments \
  +[--duration 600 --rtprio 1 --cpu-list 4,6,52,54,56,58 --cpu-main-thread 2] <1>
   I0829 13:35:22.632263       1 main.go:59] Succeeded to run the oslat command: oslat V 2.00
   Total runtime: 		600 seconds
   Thread priority: 	SCHED_FIFO:1
  
positive.code: # Thread 8: 45766 56169 56171 ...# 00015 others
  # Thread 9: 02974 08094 13214 ... # 00090 others
  ----
  <1> The docker arguments provided by the user.
  <2> The user is notified about the measured latency and the configured latency.
  <3> The arguments for the `cyclictest` command.
  <4> The maximum latencies measured on each thread.
  
  [id="cnf-performing-end-to-end-tests-running-oslat"]
  === Running oslat
  
  [source, terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_CPUS=7 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.10 /usr/bin/test-run.sh -ginkgo.focus="oslat"
  ----
  
  The above command runs the `cyclictest` tool for 10 minutes (600 seconds). The test runs successfully when the maximum observed latency is lower than `MAXIMUM_LATENCY` (20 μs), and the command line displays `SUCCESS!`.
  
  .Example failure output
  [source, terminal]
  ----
  $ docker run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e IMAGE_REGISTRY=quay.io/titzhak -e CNF_TESTS_IMAGE=cnf-tests:latest -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e DISCOVERY_MODE=true -e MAXIMUM_LATENCY=20 -e LATENCY_TEST_CPUS=7 quay.io/titzhak/cnf-tests:latest usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="oslat" <1>
  
  running /usr/bin//validationsuite -ginkgo.v -ginkgo.focus=oslat
  I0829 12:36:55.386776       8 request.go:668] Waited for 1.000303471s due to client-side throttling, not priority and fairness, request: GET:https://api.cnfdc8.t5g.lab.eng.bos.redhat.com:6443/apis/authentication.k8s.io/v1?timeout=32s
  
negative.code:      Max-Min:	 36 37 48 27 27 18 (us)
      Duration:	 599.667 599.667 599.667 599.667 599.667 599.667 (sec)
  ----
  <1> The list of CPUs running the `oslat` command. The `LATENCY_TEST_CPUS` variable providessSeven CPUs. You can only see six CPUs in total because one runs the `oslat` tool.
  <2> You can see the measured latency and the configured latency.
  <3> The maximum latency values in microseconds that each CPU measures.
  
  [id="cnf-performing-end-to-end-tests-troubleshooting_{context}"]
  == Troubleshooting
  