| 0122938a | 20-Mar-2025 |
Tomas Glozar <[email protected]> |
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla
rtla: Always set all tracer options
rtla currently only sets tracer options that are explicitly set by the user, with the exception of OSNOISE_WORKLOAD.
This leads to improper behavior in case rtla is run with those options not set to the default value. rtla does reset them to the original value upon exiting, but that does not protect it from starting with non-default values set either by an improperly exited rtla or by another user of the tracers.
For example, after running this command:
$ echo 1 > /sys/kernel/tracing/osnoise/stop_tracing_us
all runs of rtla will stop at the 1us threshold, even if not requested by the user:
$ rtla osnoise hist Index CPU-000 CPU-001 1 8 5 2 5 9 3 1 2 4 6 1 5 2 1 6 0 1 8 1 1 12 0 1 14 1 0 15 1 0 over: 0 0 count: 25 21 min: 1 1 avg: 3.68 3.05 max: 15 12 rtla osnoise hit stop tracing
Fix the problem by setting the default value for all tracer options if the user has not provided their own value.
For most of the options, it's enough to just drop the if clause checking for the value being set. For cpus, "all" is used as the default value, and for osnoise default period and runtime, default values of the osnoise_data variable in trace_osnoise.c are used.
Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <[email protected]> Reviewed-by: John Kacur <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| a8122a63 | 20-Mar-2025 |
Tomas Glozar <[email protected]> |
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options $ rtla osnoise h
rtla/osnoise: Set OSNOISE_WORKLOAD to true
If running rtla osnoise with NO_OSNOISE_WORKLOAD, it reports no samples:
$ echo NO_OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options $ rtla osnoise hist -d 10s Index over: 0 count: 0 min: 0 avg: 0 max: 0
This situation can also happen when running rtla-osnoise after an improperly exited rtla-timerlat run.
Set OSNOISE_WORKLOAD in rtla-osnoise, too, similarly to what we already did for timerlat in commit 217f0b1e990e ("rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads") and commit d8d866171a41 ("rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads").
Note that there is no user workload mode for rtla-osnoise yet, so OSNOISE_WORKLOAD is always set to true.
Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Signed-off-by: Tomas Glozar <[email protected]> Reviewed-by: John Kacur <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 20d6b075 | 20-Mar-2025 |
Tomas Glozar <[email protected]> |
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly th
rtla: Unify apply_config between top and hist
The functions osnoise_top_apply_config and osnoise_hist_apply_config, as well as timerlat_top_apply_config and timerlat_hist_apply_config, are mostly the same.
Move common part from them into separate functions osnoise_apply_config and timerlat_apply_config.
For rtla-timerlat, also unify params->user_hist and params->user_top into one field called params->user_data, and move several fields used only by timerlat-top into the top-only section of struct timerlat_params.
Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Reviewed-by: John Kacur <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 025b2179 | 20-Mar-2025 |
Tomas Glozar <[email protected]> |
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be sha
rtla/osnoise: Unify params struct
Instead of having separate structs osnoise_top_params and osnoise_hist_params, use one struct osnoise_params for both.
This allows code using the structs to be shared between osnoise-top and osnoise-hist.
Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Reviewed-by: John Kacur <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| c57c58a6 | 13-Mar-2025 |
Tomas Glozar <[email protected]> |
rtla: Fix segfault in save_trace_to_file call
Running rtla with exit on threshold, but without saving trace leads to a segmenetation fault:
$ rtla timerlat hist -T 10 ... Max timerlat IRQ latency f
rtla: Fix segfault in save_trace_to_file call
Running rtla with exit on threshold, but without saving trace leads to a segmenetation fault:
$ rtla timerlat hist -T 10 ... Max timerlat IRQ latency from idle: 4.29 us in cpu 0 Segmentation fault
This is caused by null pointer deference in the call of save_trace_to_file, which attempts to dereference an uninitialized osnoise_tool variable:
save_trace_to_file(record->trace.inst, params->trace_output); ^ this is uninitialized if params->trace_output is not set
Fix this by not attempting to dereference "record" if it is NULL and passing NULL instead. As a safety measure, the first field is also checked for NULL inside save_trace_to_file.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Costa Shulyupin <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: dc4d4e7c72d1 ("rtla: Refactor save_trace_to_file") Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| dc4d4e7c | 19-Feb-2025 |
Costa Shulyupin <[email protected]> |
rtla: Refactor save_trace_to_file
The functions osnoise_hist_main(), osnoise_top_main(), timerlat_hist_main(), and timerlat_top_main() are lengthy and contain duplicated code.
Refactor by consolida
rtla: Refactor save_trace_to_file
The functions osnoise_hist_main(), osnoise_top_main(), timerlat_hist_main(), and timerlat_top_main() are lengthy and contain duplicated code.
Refactor by consolidating the duplicate lines into the save_trace_to_file() function.
Cc: Daniel Bristot de Oliveira <[email protected]> Cc: John Kacur <[email protected]> Cc: "Luis Claudio R. Goncalves" <[email protected]> Cc: Eder Zulian <[email protected]> Cc: Dan Carpenter <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Costa Shulyupin <[email protected]> Reviewed-by: Tomas Glozar <[email protected]> Tested-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 9a82a3fd | 18-Feb-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_top: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fai
rtla/timerlat_top: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fails to load for whatever reason, rtla falls back to the old implementation.
The collection of samples using the BPF program is fully self-contained and requires no activity of the userspace part of rtla during the measurement. Thus, rtla only pulls the summary from the BPF map and displays it every second, improving the performance.
In --aa-only mode, the BPF program does not collect any data and only signalizes the end of tracing to userspace. An optimization that re-used the main trace instance for auto-analysis in aa-only mode was dropped, as rtla no longer turns tracing on in the main trace instance, making it useless for auto-analysis.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Cc: Clark Williams <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 18923806 | 18-Feb-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_top: Move divisor to update
Unlike timerlat-hist, timerlat-top applies the output divisor used to set ns/us mode when printing results instead of applying it when collecting the sample
rtla/timerlat_top: Move divisor to update
Unlike timerlat-hist, timerlat-top applies the output divisor used to set ns/us mode when printing results instead of applying it when collecting the samples.
Move the application of the divisor from timerlat_top_print into timerlat_top_update to make it consistent with timerlat-hist.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Cc: Clark Williams <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| fd7925cb | 18-Feb-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_hist: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fa
rtla/timerlat_hist: Use BPF to collect samples
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF program fails to load for whatever reason, rtla falls back to the old implementation.
The collection of samples using the BPF program is fully self-contained and requires no activity of the userspace part of rtla during the measurement. Thus, instead of waking up every second to collect samples, rtla simply sleeps until woken up by a signal or threshold overflow.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Cc: Clark Williams <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| e34293dd | 18-Feb-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat: Add BPF skeleton to collect samples
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint and collects both the summary and the histogram (if requested) into BPF map
rtla/timerlat: Add BPF skeleton to collect samples
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint and collects both the summary and the histogram (if requested) into BPF maps (one map of each kind per context).
The program is designed to be used for both timerlat-top and timerlat-hist. If using with timerlat-top, the "entries" parameter is set to zero, which prevents the BPF program from recording histogram entries. In that case, the maps for histograms do not have to be created, as the BPF verifier will identify the code using them as unreachable.
An IRQ or thread latency threshold might be supplied to stop recording if hit, similar to the timerlat tracer threshold, which stops ftrace tracing if hit. A BPF ringbuffer is used to signal threshold overflow to userspace. In aa-only mode, this is the only function of the BPF program.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Cc: Clark Williams <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 8ccd9d8b | 23-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla: Add function to report missed events
Add osnoise_report_missed_events to be used to report the number of missed events either during or after an osnoise or timerlat run. Also, display the perc
rtla: Add function to report missed events
Add osnoise_report_missed_events to be used to report the number of missed events either during or after an osnoise or timerlat run. Also, display the percentage of missed events compared to the total number of received events.
If an unknown number of missed events was reported during the run, the entire number of missed events is reported as unknown.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 2aee44f7 | 23-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla: Count all processed events
Add a field processed_events to struct trace_instance and increment it in collect_registered_events, regardless of whether a handler is registered for the event.
Th
rtla: Count all processed events
Add a field processed_events to struct trace_instance and increment it in collect_registered_events, regardless of whether a handler is registered for the event.
The purpose is to calculate the percentage of events that were missed due to tracefs buffer overflow.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| d6fcd28f | 23-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla: Count missed trace events
Add function collect_missed_events to trace.c to act as a callback for tracefs_follow_missed_events, summing the number of total missed events into a new field missin
rtla: Count missed trace events
Add function collect_missed_events to trace.c to act as a callback for tracefs_follow_missed_events, summing the number of total missed events into a new field missing_events of struct trace_instance.
In case record->missed_events is negative, trace->missed_events is set to UINT64_MAX to signify an unknown number of events was missed.
The callback is activated on initialization of the trace instance.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| b91cfd9f | 15-Jan-2025 |
Costa Shulyupin <[email protected]> |
tools/rtla: Add osnoise_trace_is_off()
All of the users of trace_is_off() passes in &record->trace as the second parameter, where record is a pointer to a struct osnoise_tool. This record could be N
tools/rtla: Add osnoise_trace_is_off()
All of the users of trace_is_off() passes in &record->trace as the second parameter, where record is a pointer to a struct osnoise_tool. This record could be NULL and there is a hidden dependency that the trace field is the first field to allow &record->trace to work with a NULL record pointer.
In order to make this code a bit more robust, as record shouldn't be dereferenced if it is NULL, even if the code does work, create a new function called osnoise_trace_is_off() that takes the pointer to a struct osnoise_tool as its second parameter. This way it can properly test if it is NULL before it dereferences it.
The old function trace_is_off() is removed and the function osnoise_trace_is_off() is added into osnoise.c which is what the struct osnoise_tool is associated with.
Cc: John Kacur <[email protected]> Cc: "Luis Claudio R. Goncalves" <[email protected]> Cc: Eder Zulian <[email protected]> Cc: Dan Carpenter <[email protected]> Cc: Tomas Glozar <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Costa Shulyupin <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 217f0b1e | 07-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/option
rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally:
$ rtla timerlat top -u ^\Quit (core dumped) $ rtla timerlat top -k -d 1s Timer Latency 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max
The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options
Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue.
Cc: [email protected] Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| d8d86617 | 07-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/optio
rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads
When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally:
$ rtla timerlat hist -u ^\Quit (core dumped) $ rtla timerlat hist -k -d 1s Index over: count: min: avg: max: ALL: IRQ Thr Usr count: 0 0 0 min: - - - avg: - - - max: - - -
The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options
Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue.
Cc: [email protected] Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: ed774f7481fa ("rtla/timerlat_hist: Add timerlat user-space support") Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| 80967b35 | 16-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_top: Abort event processing on second signal
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently le
rtla/timerlat_top: Abort event processing on second signal
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately.
This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| d6899e56 | 16-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_hist: Abort event processing on second signal
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently l
rtla/timerlat_hist: Abort event processing on second signal
If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately.
This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples.
Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| a4dfce75 | 16-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_top: Stop timerlat tracer on signal
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the m
rtla/timerlat_top: Stop timerlat tracer on signal
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs.
In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits.
Cc: [email protected] Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|
| c73cab9d | 16-Jan-2025 |
Tomas Glozar <[email protected]> |
rtla/timerlat_hist: Stop timerlat tracer on signal
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the
rtla/timerlat_hist: Stop timerlat tracer on signal
Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs.
In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits.
Cc: [email protected] Cc: John Kacur <[email protected]> Cc: Luis Goncalves <[email protected]> Cc: Gabriele Monaco <[email protected]> Link: https://lore.kernel.org/[email protected] Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <[email protected]> Signed-off-by: Steven Rostedt (Google) <[email protected]>
show more ...
|