Chapter 15. Measuring scheduling latency using rtla-osnoise in RHEL for Real Time
Chapter 15. Measuring scheduling latency using rtla-osnoise in RHEL for Real Time An ultra-low latency is an environment that is optimized to process high volumes of data packets with low tolerance for delay. Providing exclusive resources to applications, including the CPU, is a prevalent practice in ultra-low-latency environments. For example, for high performance network processing in network functions virtualization (NFV) applications, a single application has the CPU power limit set to run tasks continuously. The Linux kernel includes the real-time analysis ( rtla ) tool, which provides an interface for the operating system noise ( osnoise ) tracer. The operating system noise is the interference that occurs in an application as a result of activities inside the operating system. Linux systems can experience noise due to: Non maskable interrupts (NMIs) Interrupt requests (IRQs) Soft interrupt requests (SoftIRQs) Other system threads activity Hardware-related jobs, such as non maskable high priority system management interrupts (SMIs) 15.1. The rtla-osnoise tracer The Linux kernel includes the real-time analysis ( rtla ) tool, which provides an interface for the operating system noise ( osnoise ) tracer. The rtla-osnoise tracer creates a thread that runs periodically for a specified given period. At the start of a period , the thread disables interrupts, starts sampling, and captures the time in a loop. The rtla-osnoise tracer provides the following capabilities: Measure how much operating noise a CPU receives. Characterize the type of operating system noise occurring in the CPU. Print optimized trace reports that help to define the root cause of unexpected results. Saves an interference counter for each interference source. The interference counter for non maskable interrupts (NMIs), interrupt requests (IRQs), software interrupt requests (SoftIRQs), and threads increase when the tool detects the entry events for these interferences. The rtla-osnoise tracer prints a run report with the following information about the noise sources at the conclusion of the period: Total amount of noise. The maximum amount of noise. The percentage of CPU that is allocated to the thread. The counters for the noise sources. 15.2. Configuring the rtla-osnoise tracer to measure scheduling latency You can configure the rtla-osnoise tracer by adding osnoise in the curret_tracer file of the tracing system. The current_tracer file is generally mounted in the /sys/kernel/tracing/ directory. The rtla-osnoise tracer measures the interrupt requests (IRQs) and saves the trace output for analysis when a thread latency is more than 20 microseconds for a single noise occurrence. Procedure List the current tracer: The no operations ( nop ) is the default tracer. Add the timerlat tracer in the current_tracer file of the tracing system: Generate the tracing output: 15.3. The rtla-osnoise options for configuration The configuration options for the rtla-osnoise tracer is available in the /sys/kernel/tracing/ directory. Configuration options for rtla-osnoise osnoise/cpus Configures the CPUs for the osnoise thread to run on. osnoise/period_us Configures the period for a osnoise thread run. osnoise/runtime_us Configures the run duration for a osnoise thread. osnoise/stop_tracing_us Stops the system tracing if a single noise is more than the configured value. Setting 0 disables this option. osnoise/stop_tracing_total_us Stops the system tracing if the total noise is more than the configured value. Setting 0 disables this option. tracing_thresh Sets the minimum delta between two time() call reads to be considered as noise, in microseconds. When set to 0 , tracing_thresh uses the default value, which is 5 microseconds. 15.4. The rtla-osnoise tracepoints The rtla-osnoise includes a set of tracepoints to identify the source of the operating system noise ( osnoise ). Trace points for rtla-osnoise osnoise:sample_threshold Displays a noise when the noise is more than the configured threshold ( tolerance_ns ). osnoise:nmi_noise Displays noise and the noise duration from non maskable interrupts (NMIs). osnoise:irq_noise Displays noise and the noise duration from interrupt requests (IRQs). osnoise:softirq_noise Displays noise and the noise duration from soft interrupt requests (SoftIRQs), osnoise:thread_noise Displays noise and the noise duration from a thread. 15.5. The rtla-osnoise tracer options The osnoise/options file includes a set of on and off configuration options for the rtla-osnoise tracer. Options for rtla-osnoise DEFAULTS Resets the options to the default value. OSNOISE_WORKLOAD Stops the osnoise workload dispatch. PANIC_ON_STOP Sets the panic() call if the tracer stops. This option captures a vmcore dump file. OSNOISE_PREEMPT_DISABLE Disables preemption for osnoise workloads, which allows only interrupt requests (IRQs) and hardware-related noise. OSNOISE_IRQ_DISABLE Disables interrupt requests (IRQs) for osnoise workloads, which allows only non maskable interrupts (NMIs) and hardware-related noise. 15.6. Measuring operating system noise with the rtla-osnoise-top tracer The rtla osnoise-top tracer measures and prints a periodic summary from the osnoise tracer along with the information about the occurrence counters of the interference source. Procedure Measure the system noise: The command output displays a periodic summary with information about the real-time priority, the assigned CPUs to run the thread, and the period of the run in microseconds. 15.7. The rtla-osnoise-top tracer options By using the rtla osnoise top --help command, you can view the help usage on the available options for the rtla-osnoise-top tracer. Options for rtla-osnoise-top -a, --auto us Sets the automatic trace mode. This mode sets some commonly used options while debugging the system. It is equivalent to use -s us -T 1 and -t . -p, --period us Sets the osnoise tracer duration period in microseconds. -r, --runtime us Sets the osnoise tracer runtime in microseconds. -s, --stop us Stops the trace if a single sample is more than the argument in microseconds. With -t , the command saves the trace to the output. -S, --stop-total us Stops the trace if the total sample is more than the argument in microseconds. With -T , the command saves a trace to the output. -T, --threshold us Specifies the minimum delta between two time reads to be considered noise. The default threshold is 5 us. -q, --quiet Prints only a summary at the end of a run. -c, --cpus cpu-list Sets the osnoise tracer to run the sample threads on the assigned cpu-list . -d, --duration time[s|m|h|d] Sets the duration of a run. -D, --debug Prints debug information. -t, --trace[=file] Saves the stopped trace to [file|osnoise_trace.txt] file. -e, --event sys:event Enables an event in the trace ( -t ) session. The argument can be a specific event, for example -e sched:sched_switch , or all events of a system group, such as -e sched system group. --filter <filter> Filters the -e sys:event system event with a filter expression. --trigger <trigger> Enables a trace event trigger to the -e sys:event system event. -P, --priority o:prio|r:prio|f:prio|d:runtime:period Sets the scheduling parameters to the osnoise tracer threads. -h, --help Prints the help menu.
cat /sys/kernel/tracing/current_tracer nop cd /sys/kernel/tracing/ echo osnoise > current_tracer cat trace tracer: osnoise rtla osnoise top -P F:1 -c 0-3 -r 900000 -d 1M -q