anchor.diff:  ----
   $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e PERFORMANCE_PROFILE_MANIFEST_OVERRIDE=/kubeconfig/manifest.yaml registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh
   ----
  -
  -[id="cnf-performing-end-to-end-tests-cluster-impacts_{context}"]
  -== Impacts on the cluster
  -
  -Depending on the feature, running the test suite could cause different impacts on the cluster. In general, only the SCTP tests do not change the cluster configuration. All of the other features have various impacts on the configuration.
  -
  -[id="cnf-performing-end-to-end-tests-sctp_{context}"]
  -=== SCTP
  -
  -SCTP tests just run different pods on different nodes to check connectivity. The impacts on the cluster are related to running simple pods on two nodes.
  -
  -[id="cnf-performing-end-to-end-tests-xtu32_{context}"]
  -=== XT_U32
  -
  -XT_U32 tests run pods on different nodes to check iptables rule that utilize xt_u32. The impacts on the cluster are related to running simple pods on two nodes.
  -
  -[id="cnf-performing-end-to-end-tests-sr-iov_{context}"]
  -=== SR-IOV
  -
  -SR-IOV tests require changes in the SR-IOV network configuration, where the tests create and destroy different types of configuration.
  -
  -This might have an impact if existing SR-IOV network configurations are already installed on the cluster, because there may be conflicts depending on the priority of such configurations.
  -
  -At the same time, the result of the tests might be affected by existing configurations.
  -
  -[id="cnf-performing-end-to-end-tests-ptp_{context}"]
  -=== PTP
  -
  -PTP tests apply a PTP configuration to a set of nodes of the cluster. As with SR-IOV, this might conflict with any existing PTP configuration already in place, with unpredictable results.
  -
  -[id="cnf-performing-end-to-end-tests-performance_{context}"]
  -=== Performance
  -
  -Performance tests apply a performance profile to the cluster. The effect of this is changes in the node configuration, reserving CPUs, allocating memory huge pages, and setting the kernel packages to be realtime. If an existing profile named `performance` is already available on the cluster, the tests do not deploy it.
  -
  -[id="cnf-performing-end-to-end-tests-dpdk_{context}"]
  -=== DPDK
  -
  -DPDK relies on both performance and SR-IOV features, so the test suite configures both a performance profile and SR-IOV networks, so the impacts are the same as those described in SR-IOV testing and performance testing.
  -
  -[id="cnf-performing-end-to-end-tests-container-mount-namespace_{context}"]
  -=== Container-mount-namespace
  -
  -The validation test for `container-mount-namespace` mode only checks that the appropriate `MachineConfig` objects are present and active, and has no additional impact on the node.
  -
  -[id="cnf-performing-end-to-end-tests-cleaning-up_{context}"]
  -=== Cleaning up
  -
  -After running the test suite, all the dangling resources are cleaned up.
positive.code: <1> The list of CPUs running the `oslat` command. Seven CPUs are provided through the `LATENCY_TEST_CPUS` variable. Only six CPUs are displayed in total because one is used to run the `oslat` tool.
  <2> The user is notified about the measured latency and the configured latency.
  <3> The maximum latency values in microseconds that are measured on each CPU.
  
  [id="cnf-performing-end-to-end-tests-image-parameters_{context}"]
  == Image parameters
  
  Depending on the requirements, the tests can use different images. There are two images used by the tests that can be changed using the following environment variables:
  
  * `CNF_TESTS_IMAGE`
  * `DPDK_TESTS_IMAGE`
  
  For example, to change the `CNF_TESTS_IMAGE` with a custom registry run the following command:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e CNF_TESTS_IMAGE="custom-cnf-tests-image:latests" registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-ginko-parameters_{context}"]
  === Ginkgo parameters
  
  The test suite is built upon the ginkgo BDD framework. This means that it accepts parameters for filtering or skipping tests.
  
  You can use the `-ginkgo.focus` parameter to filter a set of tests:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.focus="performance|sctp"
  ----
  
  You can run only the latency test using the `-ginkgo.focus` parameter.
  
  To run only the latency test, you must provide the `-ginkgo.focus` parameter and the `PERF_TEST_PROFILE` environment variable that contains the name of the performance profile that needs to be tested. For example:
  
  [source,terminal]
  ----
  $ docker run --rm -v $KUBECONFIG:/kubeconfig -e KUBECONFIG=/kubeconfig -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e OSLAT_MAXIMUM_LATENCY=20 -e PERF_TEST_PROFILE=<performance_profile_name> registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.focus="\[performance\]\[config\]|\[performance\]\ Latency\ Test"
  ----
  
  [NOTE]
  ====
  There is a particular test that requires both SR-IOV and SCTP. Given the selective nature of the `focus` parameter, this test is triggered by only placing the `sriov` matcher. If the tests are executed against a cluster where SR-IOV is installed but SCTP is not, adding the `-ginkgo.skip=SCTP` parameter causes the tests to skip SCTP testing.
  ====
  
  [id="cnf-performing-end-to-end-tests-available-features_{context}"]
  === Available features
  
  The set of available features to filter are:
  
  * `performance`
  * `sriov`
  * `ptp`
  * `sctp`
  * `xt_u32`
  * `dpdk`
  * `container-mount-namespace`
  
  [id="cnf-performing-end-to-end-tests-dry-run_{context}"]
  == Dry run
  
  Use this command to run in dry-run mode. This is useful for checking what is in the test suite and provides output for all of the tests the image would run.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.dryRun -ginkgo.v
  ----
  
  [id="cnf-performing-end-to-end-tests-disconnected-mode_{context}"]
  == Disconnected mode
  
  The CNF tests image support running tests in a disconnected cluster, meaning a cluster that is not able to reach outer registries. This is done in two steps:
  
  . Performing the mirroring.
  
  . Instructing the tests to consume the images from a custom registry.
  
  [id="cnf-performing-end-to-end-tests-mirroring-images-to-custom-registry_{context}"]
  === Mirroring the images to a custom registry accessible from the cluster
  
  A `mirror` executable is shipped in the image to provide the input required by `oc` to mirror the images needed to run the tests to a local registry.
  
  Run this command from an intermediate machine that has access both to the cluster and to link:https://catalog.redhat.com/software/containers/explore[registry.redhat.io] over the internet:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror -registry my.local.registry:5000/ |  oc image mirror -f -
  ----
  
  Then, follow the instructions in the following section about overriding the registry used to fetch the images.
  
  [id="instruct-the-tests-to-consume-images-from-a-custom-registry_{context}"]
  === Instruct the tests to consume those images from a custom registry
  
  This is done by setting the `IMAGE_REGISTRY` environment variable:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e IMAGE_REGISTRY="my.local.registry:5000/" -e CNF_TESTS_IMAGE="custom-cnf-tests-image:latests" registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-mirroring-to-cluster-internal-registry_{context}"]
  === Mirroring to the cluster internal registry
  
  {product-title} provides a built-in container image registry, which runs as a standard workload on the cluster.
  
  .Procedure
  
  . Gain external access to the registry by exposing it with a route:
  +
  [source,terminal]
  ----
  $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
  ----
  
  . Fetch the registry endpoint:
  +
  [source,terminal]
  ----
  REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
  ----
  
  . Create a namespace for exposing the images:
  +
  [source,terminal]
  ----
  $ oc create ns cnftests
  ----
  
  . Make that image stream available to all the namespaces used for tests. This is required to allow the tests namespaces to fetch the images from the `cnftests` image stream.
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:sctptest:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:cnf-features-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:performance-addon-operators-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:dpdk-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:sriov-conformance-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:xt-u32-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:vrf-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:gatekeeper-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:ovs-qos-testing:default --namespace=cnftests
  ----
  
  . Retrieve the docker secret name and auth token:
  +
  [source,bash]
  ----
  SECRET=$(oc -n cnftests get secret | grep builder-docker | awk {'print $1'}
  TOKEN=$(oc -n cnftests get secret $SECRET -o jsonpath="{.data['\.dockercfg']}" | base64 --decode | jq '.["image-registry.openshift-image-registry.svc:5000"].auth')
  ----
  
  . Write a `dockerauth.json` similar to this:
  +
  [source,bash]
  ----
  echo "{\"auths\": { \"$REGISTRY\": { \"auth\": $TOKEN } }}" > dockerauth.json
  ----
  
  . Do the mirroring:
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror -registry $REGISTRY/cnftests |  oc image mirror --insecure=true -a=$(pwd)/dockerauth.json -f -
  ----
  
  . Run the tests:
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e IMAGE_REGISTRY=image-registry.openshift-image-registry.svc:5000/cnftests cnf-tests-local:latest /usr/bin/test-run.sh
  ----
  
  [id="mirroring-different-set-of-images_{context}"]
  === Mirroring a different set of images
  
  .Procedure
  
  . The `mirror` command tries to mirror the u/s images by default. This can be overridden by passing a file with the following format to the image:
  +
  [source,yaml]
  ----
  [
      {
          "registry": "public.registry.io:5000",
          "image": "imageforcnftests:4.8"
      },
      {
          "registry": "public.registry.io:5000",
          "image": "imagefordpdk:4.8"
      }
  ]
  ----
  
  . Pass it to the `mirror` command, for example saving it locally as `images.json`. With the following command, the local path is mounted in `/kubeconfig` inside the container and that can be passed to the mirror command.
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror --registry "my.local.registry:5000/" --images "/kubeconfig/images.json" |  oc image mirror -f -
  ----
  
  [id="discovery-mode_{context}"]
  == Discovery mode
  
  Discovery mode allows you to validate the functionality of a cluster without altering its configuration. Existing environment configurations are used for the tests. The tests attempt to find the configuration items needed and use those items to execute the tests. If resources needed to run a specific test are not found, the test is skipped, providing an appropriate message to the user. After the tests are finished, no cleanup of the pre-configured configuration items is done, and the test environment can be immediately used for another test run.
  
  Some configuration items are still created by the tests. These are specific items needed for a test to run; for example, a SR-IOV Network. These configuration items are created in custom namespaces and are cleaned up after the tests are executed.
  
  An additional bonus is a reduction in test run times. As the configuration items are already there, no time is needed for environment configuration and stabilization.
  
  To enable discovery mode, the tests must be instructed by setting the `DISCOVERY_MODE` environment variable as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  DISCOVERY_MODE=true registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="required-environment-config-prereqs_{context}"]
  === Required environment configuration prerequisites
  
  .SR-IOV tests
  
  Most SR-IOV tests require the following resources:
  
  * `SriovNetworkNodePolicy`.
  * At least one with the resource specified by `SriovNetworkNodePolicy` being allocatable; a resource count of at least 5 is considered sufficient.
  
  Some tests have additional requirements:
  
  * An unused device on the node with available policy resource, with link state `DOWN` and not a bridge slave.
  * A `SriovNetworkNodePolicy` with a MTU value of `9000`.
  
  .DPDK tests
  
  The DPDK related tests require:
  
  * A performance profile.
  * A SR-IOV policy.
  * A node with resources available for the SR-IOV policy and available with the `PerformanceProfile` node selector.
  
  .PTP tests
  
  * A slave `PtpConfig` (`ptp4lOpts="-s" ,phc2sysOpts="-a -r"`).
  * A node with a label matching the slave `PtpConfig`.
  
  .SCTP tests
  
  * `SriovNetworkNodePolicy`.
  * A node matching both the `SriovNetworkNodePolicy` and a `MachineConfig` that enables SCTP.
  
  .XT_U32 tests
  
  * A node with a machine config that enables XT_U32.
  
  .Performance Operator tests
  
  Various tests have different requirements. Some of them are:
  
  * A performance profile.
  * A performance profile having `profile.Spec.CPU.Isolated = 1`.
  * A performance profile having `profile.Spec.RealTimeKernel.Enabled == true`.
  * A node with no huge pages usage.
  
  .Container-mount-namespace tests
  
  * A node with a machine config which enables `container-mount-namespace` mode
  
  [id="limiting-nodes-used-during-tests_{context}"]
  === Limiting the nodes used during tests
  
  The nodes on which the tests are executed can be limited by specifying a `NODES_SELECTOR` environment variable. Any resources created by the test are then limited to the specified nodes.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  NODES_SELECTOR=node-role.kubernetes.io/worker-cnf registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="using-single-performance-profile_{context}"]
  === Using a single performance profile
  
  The resources needed by the DPDK tests are higher than those required by the performance test suite. To make the execution faster, the performance profile used by tests can be overridden using one that also serves the DPDK test suite.
  
  To do this, a profile like the following one can be mounted inside the container, and the performance tests can be instructed to deploy it.
  
  [source,yaml]
  ----
  apiVersion: performance.openshift.io/v2
  kind: PerformanceProfile
  metadata:
    name: performance
  spec:
    cpu:
      isolated: "4-15"
      reserved: "0-3"
    hugepages:
      defaultHugepagesSize: "1G"
      pages:
      - size: "1G"
        count: 16
        node: 0
    realTimeKernel:
      enabled: true
    nodeSelector:
      node-role.kubernetes.io/worker-cnf: ""
  ----
  
  [NOTE]
  ====
  When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.
  ====
  
  To override the performance profile used, the manifest must be mounted inside the container and the tests must be instructed by setting the `PERFORMANCE_PROFILE_MANIFEST_OVERRIDE` parameter as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  PERFORMANCE_PROFILE_MANIFEST_OVERRIDE=/kubeconfig/manifest.yaml registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="disabling-performance-profile-cleanup_{context}"]
  === Disabling the performance profile cleanup
  
  When not running in discovery mode, the suite cleans up all the created artifacts and configurations. This includes the performance profile.
  
  When deleting the performance profile, the machine config pool is modified and nodes are rebooted. After a new iteration, a new profile is created. This causes long test cycles between runs.
  
  To speed up this process, set `CLEAN_PERFORMANCE_PROFILE="false"` to instruct the tests not to clean the performance profile. In this way, the next iteration will not need to create it and wait for it to be applied.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  CLEAN_PERFORMANCE_PROFILE="false" registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-running-in-single-node-cluster_{context}"]
  == Running in a single node cluster
  
  Running tests on a single node cluster causes the following limitations to be imposed:
  
  * Longer timeouts for certain tests, including SR-IOV and SCTP tests
  * Tests requiring master and worker nodes are skipped
  
  Longer timeouts concern SR-IOV and SCTP tests. Reconfiguration requiring node reboots cause a reboot of the entire environment, including the OpenShift control plane, and therefore takes longer to complete. All PTP tests requiring a master and worker node are skipped. No additional configuration is needed because the tests check for the number of nodes at startup and adjust test behavior accordingly.
  
  PTP tests can run in Discovery mode. The tests look for a PTP master configured outside of the cluster.
  
  For more information, see the Discovery mode section.
  // TODO update to xref
  
  To enable Discovery mode, the tests must be instructed by setting the `DISCOVERY_MODE` environment variable as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  DISCOVERY_MODE=true registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [discrete]
  === Required parameters
  
  * `ROLE_WORKER_CNF=master` - Required because master is the only machine pool to which the node will belong.
  * `XT_U32TEST_HAS_NON_CNF_WORKERS=false` - Required to instruct the xt_u32 negative test to skip because there are only nodes where the module is loaded.
  * `SCTPTEST_HAS_NON_CNF_WORKERS=false` - Required to instruct the SCTP negative test to skip because there are only nodes where the module is loaded.
  
  [id="cnf-performing-end-to-end-tests-troubleshooting_{context}"]
  == Troubleshooting
  
negative.code: ====
  Currently, not all tests run selectively on the nodes belonging to the pool.
  ====
  
  [id="cnf-performing-end-to-end-tests-dry-run_{context}"]
  == Dry run
  
  Use this command to run in dry-run mode. This is useful for checking what is in the test suite and provides output for all of the tests the image would run.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.dryRun -ginkgo.v
  ----
  
  [id="cnf-performing-end-to-end-tests-disconnected-mode_{context}"]
  == Disconnected mode
  
  The CNF tests image support running tests in a disconnected cluster, meaning a cluster that is not able to reach outer registries. This is done in two steps:
  
  . Performing the mirroring.
  
  . Instructing the tests to consume the images from a custom registry.
  
  [id="cnf-performing-end-to-end-tests-mirroring-images-to-custom-registry_{context}"]
  === Mirroring the images to a custom registry accessible from the cluster
  
  A `mirror` executable is shipped in the image to provide the input required by `oc` to mirror the images needed to run the tests to a local registry.
  
  Run this command from an intermediate machine that has access both to the cluster and to link:https://catalog.redhat.com/software/containers/explore[registry.redhat.io] over the internet:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror -registry my.local.registry:5000/ |  oc image mirror -f -
  ----
  
  Then, follow the instructions in the following section about overriding the registry used to fetch the images.
  
  [id="instruct-the-tests-to-consume-images-from-a-custom-registry_{context}"]
  === Instruct the tests to consume those images from a custom registry
  
  This is done by setting the `IMAGE_REGISTRY` environment variable:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e IMAGE_REGISTRY="my.local.registry:5000/" -e CNF_TESTS_IMAGE="custom-cnf-tests-image:latests" registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-mirroring-to-cluster-internal-registry_{context}"]
  === Mirroring to the cluster internal registry
  
  {product-title} provides a built-in container image registry, which runs as a standard workload on the cluster.
  
  .Procedure
  
  . Gain external access to the registry by exposing it with a route:
  +
  [source,terminal]
  ----
  $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
  ----
  
  . Fetch the registry endpoint:
  +
  [source,terminal]
  ----
  REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
  ----
  
  . Create a namespace for exposing the images:
  +
  [source,terminal]
  ----
  $ oc create ns cnftests
  ----
  
  . Make that image stream available to all the namespaces used for tests. This is required to allow the tests namespaces to fetch the images from the `cnftests` image stream.
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:sctptest:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:cnf-features-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:performance-addon-operators-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:dpdk-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:sriov-conformance-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:xt-u32-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:vrf-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:gatekeeper-testing:default --namespace=cnftests
  ----
  +
  [source,terminal]
  ----
  $ oc policy add-role-to-user system:image-puller system:serviceaccount:ovs-qos-testing:default --namespace=cnftests
  ----
  
  . Retrieve the docker secret name and auth token:
  +
  [source,bash]
  ----
  SECRET=$(oc -n cnftests get secret | grep builder-docker | awk {'print $1'}
  TOKEN=$(oc -n cnftests get secret $SECRET -o jsonpath="{.data['\.dockercfg']}" | base64 --decode | jq '.["image-registry.openshift-image-registry.svc:5000"].auth')
  ----
  
  . Write a `dockerauth.json` similar to this:
  +
  [source,bash]
  ----
  echo "{\"auths\": { \"$REGISTRY\": { \"auth\": $TOKEN } }}" > dockerauth.json
  ----
  
  . Do the mirroring:
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror -registry $REGISTRY/cnftests |  oc image mirror --insecure=true -a=$(pwd)/dockerauth.json -f -
  ----
  
  . Run the tests:
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e IMAGE_REGISTRY=image-registry.openshift-image-registry.svc:5000/cnftests cnf-tests-local:latest /usr/bin/test-run.sh
  ----
  
  [id="mirroring-different-set-of-images_{context}"]
  === Mirroring a different set of images
  
  .Procedure
  
  . The `mirror` command tries to mirror the u/s images by default. This can be overridden by passing a file with the following format to the image:
  +
  [source,yaml]
  ----
  [
      {
          "registry": "public.registry.io:5000",
          "image": "imageforcnftests:4.8"
      },
      {
          "registry": "public.registry.io:5000",
          "image": "imagefordpdk:4.8"
      }
  ]
  ----
  
  . Pass it to the `mirror` command, for example saving it locally as `images.json`. With the following command, the local path is mounted in `/kubeconfig` inside the container and that can be passed to the mirror command.
  +
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/mirror --registry "my.local.registry:5000/" --images "/kubeconfig/images.json" |  oc image mirror -f -
  ----
  
  [id="cnf-performing-end-to-end-tests-running-in-single-node-cluster_{context}"]
  == Running in a single node cluster
  
  Running tests on a single node cluster causes the following limitations to be imposed:
  
  * Longer timeouts for certain tests, including SR-IOV and SCTP tests
  * Tests requiring master and worker nodes are skipped
  
  Longer timeouts concern SR-IOV and SCTP tests. Reconfiguration requiring node reboots cause a reboot of the entire environment, including the OpenShift control plane, and therefore takes longer to complete. All PTP tests requiring a master and worker node are skipped. No additional configuration is needed because the tests check for the number of nodes at startup and adjust test behavior accordingly.
  
  PTP tests can run in Discovery mode. The tests look for a PTP master configured outside of the cluster.
  
  For more information, see the Discovery mode section.
  // TODO update to xref
  
  To enable Discovery mode, the tests must be instructed by setting the `DISCOVERY_MODE` environment variable as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  DISCOVERY_MODE=true registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [discrete]
  === Required parameters
  
  * `ROLE_WORKER_CNF=master` - Required because master is the only machine pool to which the node will belong.
  * `XT_U32TEST_HAS_NON_CNF_WORKERS=false` - Required to instruct the xt_u32 negative test to skip because there are only nodes where the module is loaded.
  * `SCTPTEST_HAS_NON_CNF_WORKERS=false` - Required to instruct the SCTP negative test to skip because there are only nodes where the module is loaded.
  
  [id="cnf-performing-end-to-end-tests-cluster-impacts_{context}"]
  == Impact of tests on the cluster
  
  Depending on the feature, running the test suite could cause different impacts on the cluster. In general, only the SCTP tests do not change the cluster configuration. All of the other features have various impacts on the configuration.
  
  [id="cnf-performing-end-to-end-tests-sctp_{context}"]
  === SCTP
  
  SCTP tests just run different pods on different nodes to check connectivity. The impacts on the cluster are related to running simple pods on two nodes.
  
  [id="cnf-performing-end-to-end-tests-xtu32_{context}"]
  === XT_U32
  
  XT_U32 tests run pods on different nodes to check iptables rule that utilize xt_u32. The impacts on the cluster are related to running simple pods on two nodes.
  
  [id="cnf-performing-end-to-end-tests-sr-iov_{context}"]
  === SR-IOV
  
  SR-IOV tests require changes in the SR-IOV network configuration, where the tests create and destroy different types of configuration.
  
  This might have an impact if existing SR-IOV network configurations are already installed on the cluster, because there may be conflicts depending on the priority of such configurations.
  
  At the same time, the result of the tests might be affected by existing configurations.
  
  [id="cnf-performing-end-to-end-tests-ptp_{context}"]
  === PTP
  
  PTP tests apply a PTP configuration to a set of nodes of the cluster. As with SR-IOV, this might conflict with any existing PTP configuration already in place, with unpredictable results.
  
  [id="cnf-performing-end-to-end-tests-performance_{context}"]
  === Performance
  
  Performance tests apply a performance profile to the cluster. The effect of this is changes in the node configuration, reserving CPUs, allocating memory huge pages, and setting the kernel packages to be realtime. If an existing profile named `performance` is already available on the cluster, the tests do not deploy it.
  
  [id="cnf-performing-end-to-end-tests-dpdk_{context}"]
  === DPDK
  
  DPDK relies on both performance and SR-IOV features, so the test suite configures both a performance profile and SR-IOV networks, so the impacts are the same as those described in SR-IOV testing and performance testing.
  
  [id="cnf-performing-end-to-end-tests-container-mount-namespace_{context}"]
  === Container-mount-namespace
  
  The validation test for `container-mount-namespace` mode only checks that the appropriate `MachineConfig` objects are present and active, and has no additional impact on the node.
  
  [id="cnf-performing-end-to-end-tests-cleaning-up_{context}"]
  === Cleaning up
  
  After running the test suite, all the dangling resources are cleaned up.
  
  [id="cnf-performing-end-to-end-tests-image-parameters_{context}"]
  == Override test image parameters
  
  Depending on the requirements, the tests can use different images. There are two images used by the tests that can be changed using the following environment variables:
  
  * `CNF_TESTS_IMAGE`
  * `DPDK_TESTS_IMAGE`
  
  For example, to change the `CNF_TESTS_IMAGE` with a custom registry run the following command:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig -e CNF_TESTS_IMAGE="custom-cnf-tests-image:latests" registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-ginko-parameters_{context}"]
  === Ginkgo parameters
  
  The test suite is built upon the ginkgo BDD framework. This means that it accepts parameters for filtering or skipping tests.
  
  You can use the `-ginkgo.focus` parameter to filter a set of tests:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.focus="performance|sctp"
  ----
  
  You can run only the latency test using the `-ginkgo.focus` parameter.
  
  To run only the latency test, you must provide the `-ginkgo.focus` parameter and the `PERF_TEST_PROFILE` environment variable that contains the name of the performance profile that needs to be tested. For example:
  
  [source,terminal]
  ----
  $ docker run --rm -v $KUBECONFIG:/kubeconfig -e KUBECONFIG=/kubeconfig -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=600 -e OSLAT_MAXIMUM_LATENCY=20 -e PERF_TEST_PROFILE=<performance_profile_name> registry.redhat.io/openshift4/cnf-tests-rhel8:v4.8 /usr/bin/test-run.sh -ginkgo.focus="\[performance\]\[config\]|\[performance\]\ Latency\ Test"
  ----
  
  [NOTE]
  ====
  There is a particular test that requires both SR-IOV and SCTP. Given the selective nature of the `focus` parameter, this test is triggered by only placing the `sriov` matcher. If the tests are executed against a cluster where SR-IOV is installed but SCTP is not, adding the `-ginkgo.skip=SCTP` parameter causes the tests to skip SCTP testing.
  ====
  
  [id="cnf-performing-end-to-end-tests-available-features_{context}"]
  === Available features
  
  The set of available features to filter are:
  
  * `performance`
  * `sriov`
  * `ptp`
  * `sctp`
  * `xt_u32`
  * `dpdk`
  * `container-mount-namespace`
  
  [id="discovery-mode_{context}"]
  == Discovery mode
  
  Discovery mode allows you to validate the functionality of a cluster without altering its configuration. Existing environment configurations are used for the tests. The tests attempt to find the configuration items needed and use those items to execute the tests. If resources needed to run a specific test are not found, the test is skipped, providing an appropriate message to the user. After the tests are finished, no cleanup of the pre-configured configuration items is done, and the test environment can be immediately used for another test run.
  
  Some configuration items are still created by the tests. These are specific items needed for a test to run; for example, a SR-IOV Network. These configuration items are created in custom namespaces and are cleaned up after the tests are executed.
  
  An additional bonus is a reduction in test run times. As the configuration items are already there, no time is needed for environment configuration and stabilization.
  
  To enable discovery mode, the tests must be instructed by setting the `DISCOVERY_MODE` environment variable as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  DISCOVERY_MODE=true registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="required-environment-config-prereqs_{context}"]
  === Required environment configuration prerequisites
  
  .SR-IOV tests
  
  Most SR-IOV tests require the following resources:
  
  * `SriovNetworkNodePolicy`.
  * At least one with the resource specified by `SriovNetworkNodePolicy` being allocatable; a resource count of at least 5 is considered sufficient.
  
  Some tests have additional requirements:
  
  * An unused device on the node with available policy resource, with link state `DOWN` and not a bridge slave.
  * A `SriovNetworkNodePolicy` with a MTU value of `9000`.
  
  .DPDK tests
  
  The DPDK related tests require:
  
  * A performance profile.
  * A SR-IOV policy.
  * A node with resources available for the SR-IOV policy and available with the `PerformanceProfile` node selector.
  
  .PTP tests
  
  * A slave `PtpConfig` (`ptp4lOpts="-s" ,phc2sysOpts="-a -r"`).
  * A node with a label matching the slave `PtpConfig`.
  
  .SCTP tests
  
  * `SriovNetworkNodePolicy`.
  * A node matching both the `SriovNetworkNodePolicy` and a `MachineConfig` that enables SCTP.
  
  .XT_U32 tests
  
  * A node with a machine config that enables XT_U32.
  
  .Performance Operator tests
  
  Various tests have different requirements. Some of them are:
  
  * A performance profile.
  * A performance profile having `profile.Spec.CPU.Isolated = 1`.
  * A performance profile having `profile.Spec.RealTimeKernel.Enabled == true`.
  * A node with no huge pages usage.
  
  .Container-mount-namespace tests
  
  * A node with a machine config which enables `container-mount-namespace` mode
  
  [id="limiting-nodes-used-during-tests_{context}"]
  === Limiting the nodes used during tests
  
  The nodes on which the tests are executed can be limited by specifying a `NODES_SELECTOR` environment variable. Any resources created by the test are then limited to the specified nodes.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  NODES_SELECTOR=node-role.kubernetes.io/worker-cnf registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="using-single-performance-profile_{context}"]
  === Using a single performance profile
  
  The resources needed by the DPDK tests are higher than those required by the performance test suite. To make the execution faster, the performance profile used by tests can be overridden using one that also serves the DPDK test suite.
  
  To do this, a profile like the following one can be mounted inside the container, and the performance tests can be instructed to deploy it.
  
  [source,yaml]
  ----
  apiVersion: performance.openshift.io/v2
  kind: PerformanceProfile
  metadata:
    name: performance
  spec:
    cpu:
      isolated: "4-15"
      reserved: "0-3"
    hugepages:
      defaultHugepagesSize: "1G"
      pages:
      - size: "1G"
        count: 16
        node: 0
    realTimeKernel:
      enabled: true
    nodeSelector:
      node-role.kubernetes.io/worker-cnf: ""
  ----
  
  [NOTE]
  ====
  When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.
  ====
  
  To override the performance profile used, the manifest must be mounted inside the container and the tests must be instructed by setting the `PERFORMANCE_PROFILE_MANIFEST_OVERRIDE` parameter as follows:
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  PERFORMANCE_PROFILE_MANIFEST_OVERRIDE=/kubeconfig/manifest.yaml registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="disabling-performance-profile-cleanup_{context}"]
  === Disabling the performance profile cleanup
  
  When not running in discovery mode, the suite cleans up all the created artifacts and configurations. This includes the performance profile.
  
  When deleting the performance profile, the machine config pool is modified and nodes are rebooted. After a new iteration, a new profile is created. This causes long test cycles between runs.
  
  To speed up this process, set `CLEAN_PERFORMANCE_PROFILE="false"` to instruct the tests not to clean the performance profile. In this way, the next iteration will not need to create it and wait for it to be applied.
  
  [source,terminal]
  ----
  $ docker run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig -e
  CLEAN_PERFORMANCE_PROFILE="false" registry.redhat.io/openshift-kni/cnf-tests /usr/bin/test-run.sh
  ----
  
  [id="cnf-performing-end-to-end-tests-running-the-tests_{context}"]
  == Running the latency tests
  