On s390 the perf test case ftrace sometimes fails as follows:
# ./perf test ftrace
79: perf ftrace tests : FAILED!
#
The failure depends on the kernel .config file. Some configurations
always work fine, some do not. The ftrace profile test mostly fails,
because the ring buffer was not large enough, and some lines
(especially the interesting ones with nanosleep in it) where dropped.
To achieve success for all tested kernel configurations, enlarge
the buffer to store the traces completely without wrapping.
The default buffer size is too small for all kernel configurations.
Set the buffer size of for the ftrace profile test to 16 MB.
Output after:
# ./perf test ftrace
79: perf ftrace tests : Ok
#
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: agordeev@linux.ibm.com
Cc: gor@linux.ibm.com
Cc: hca@linux.ibm.com
Cc: sumanthk@linux.ibm.com
Link: https://lore.kernel.org/r/20241119064856.641446-1-tmricht@linux.ibm.com
Suggested-by: Sven Schnelle <svens@linux.ibm.com>
Suggested-by: Namhyung Kim <namhyung@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The purpose of this test is to test for races in the exit of 'perf
trace' missing the last events, it was failing when the COMM wasn't
resolved either because we missed some PERF_RECORD_COMM or somehow
raced on getting it from procfs.
Add --no-comm to the 'perf trace' command line so that we get a
consistent, pid only output, which allows the test to achieve its goal.
This is the output from
'perf trace --no-comm -e syscalls:sys_enter_exit_group':
0.000 21953 syscalls:sys_enter_exit_group()
0.000 21955 syscalls:sys_enter_exit_group()
0.000 21957 syscalls:sys_enter_exit_group()
0.000 21959 syscalls:sys_enter_exit_group()
0.000 21961 syscalls:sys_enter_exit_group()
0.000 21963 syscalls:sys_enter_exit_group()
0.000 21965 syscalls:sys_enter_exit_group()
0.000 21967 syscalls:sys_enter_exit_group()
0.000 21969 syscalls:sys_enter_exit_group()
0.000 21971 syscalls:sys_enter_exit_group()
Now it passes:
root@number:~# perf test "trace exit race"
110: perf trace exit race : Ok
root@number:~#
root@number:~# perf test -v "trace exit race"
110: perf trace exit race : Ok
root@number:~#
If we artificially make it run just 9 times instead of the 10 it runs,
i.e. by manually doing:
trace_shutdown_race() {
for _ in $(seq 9); do
that 9 is $iter, 10 in the patch, we get:
root@number:~# vim ~acme/libexec/perf-core/tests/shell/trace_exit_race.sh
root@number:~# perf test -v "trace exit race"
--- start ---
test child forked, pid 24629
Missing output, expected 10 but only got 9
---- end(-1) ----
110: perf trace exit race : FAILED!
root@number:~#
I.e. 9 'perf trace' calls produced the expected output, the inverse grep
didn't show anything, so the patch provided by Howard for the previous
patch kicks in and shows a more informative message.
Tested-by: Howard Chu <howardchu95@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Benjamin Peterson <benjamin@engflow.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/lkml/ZzdknoHqrJbojb6P@x1
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
If it fails we need to check what was the reason, what were the lines
that didn't match the expected format, so:
root@number:~# perf test -v "trace exit race"
--- start ---
test child forked, pid 2028724
Lines not matching the expected regexp: ' +[0-9]+\.[0-9]+ +true/[0-9]+ syscalls:sys_enter_exit_group\(\)$':
0.000 :2028750/2028750 syscalls:sys_enter_exit_group()
---- end(-1) ----
110: perf trace exit race : FAILED!
root@number:~#
In this case we're not resolving the process COMM for some reason and
fallback to printing just the pid/tid, this will be fixed in a followup
patch.
Howard Chu spotted a problem with single code surrounding a regexp, that
made the test always fail, but since there were some failures when I
tested (COMM not being resolved in some of the results) the end inverse
grep would show some lines and thus didn't notice the single quote
problem.
He also provided a patch to test if less than the number of expected
matches took place but all of them with the expected output, in which
case the inverse grep wouldn't show anything, confusing the tester.
Reviewed-by: Howard Chu <howardchu95@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Benjamin Peterson <benjamin@engflow.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/lkml/ZzdknoHqrJbojb6P@x1
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
During the parallel testing, I've noticed some ftrace test failures. It
seems the regex pattern checks 100 msec of nanosleep with the error
range of 10 msec. But sometimes it's affected by other processes and
resulted in more time in the syscall.
The following output shows that it took more than 120 msec and failed.
Let's update the regex pattern so that it can allow more drifts.
perf ftrace profile test
# Total (us) Avg (us) Max (us) Count Function
121279.500 121279.500 121279.500 1 __x64_sys_clock_nanosleep
121278.400 121278.400 121278.400 1 common_nsleep
121277.800 121277.800 121277.800 1 hrtimer_nanosleep
121277.100 121277.100 121277.100 1 do_nanosleep
341760.289 56960.048 121273.400 6 schedule
176.200 25.171 31.616 7 scheduler_tick
0.923 0.923 0.923 1 native_smp_send_reschedule
345522.360 69104.472 345320.600 5 __x64_sys_execve
345486.585 69097.317 345312.700 5 do_execveat_common.isra.0
340730.300 340730.300 340730.300 1 bprm_execve
1.758 0.879 0.883 2 sched_mm_cid_before_execve
1.112 1.112 1.112 1 sched_mm_cid_after_execve
---- end(-1) ----
81: perf ftrace tests : FAILED!
Reviewed-by: Ian Rogers <irogers@google.com>
Link: https://lore.kernel.org/r/20241102231702.2262258-1-namhyung@kernel.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The cpu-list part of this testcase has proven itself to be unreliable.
Sometimes, we get "<not counted>" for system.slice when pinned to CPUs
0 and 1. In such case, the test fails.
Since we cannot simply guarantee that any system.slice load will run
on any arbitrary list of CPUs, except the whole set of all CPUs, let's
rather remove the cpu-list subtest.
Fixes: a84260e314 ("perf test stat_bpf_counters_cgrp: Enhance perf stat cgroup BPF counter test")
Signed-off-by: Michael Petlan <mpetlan@redhat.com>
Cc: vmolnaro@redhat.com
Cc: bpf@vger.kernel.org
Link: https://lore.kernel.org/r/20241101102812.576425-1-mpetlan@redhat.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The exclude_guest in the event attribute is to limit profiling in the
host environment. But I'm not sure why we want to set it by default
cause we don't care about it in most cases and I feel like it just
makes new PMU implementation complicated.
Of course it's useful for perf kvm command so I added the
exclude_GH_default variable to preserve the old behavior for perf kvm
and other commands like perf record and stat won't set the exclude bit.
This is helpful for AMD IBS case since having exclude_guest bit will
clear new feature bit due to the missing feature check logic.
$ sysctl kernel.perf_event_paranoid
kernel.perf_event_paranoid = 0
$ perf record -W -e ibs_op// -vv true 2>&1 | grep switching
switching off PERF_FORMAT_LOST support
switching off weight struct support
switching off bpf_event
switching off ksymbol
switching off cloexec flag
switching off mmap2
switching off exclude_guest, exclude_host
Intestingly, I found it sets the exclude_bit if "u" modifier is used.
I don't know why but it's neither intuitive nor consistent. Let's
remove the bit there too.
Reviewed-by: Ian Rogers <irogers@google.com>
Reviewed-by: James Clark <james.clark@linaro.org>
Reviewed-by: Ravi Bangoria <ravi.bangoria@amd.com>
Acked-by: Kan Liang <kan.liang@linux.intel.com>
Cc: James Clark <james.clark@arm.com>
Cc: Atish Patra <atishp@atishpatra.org>
Cc: Mingwei Zhang <mizhang@google.com>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Cc: Palmer Dabbelt <palmer@rivosinc.com>
Link: https://lore.kernel.org/r/20241016062359.264929-3-namhyung@kernel.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Perf test case 84 'perf pipe recording and injection test'
sometime fails on s390, especially on z/VM virtual machines.
This is caused by a very short run time of workload
# perf test -w noploop
which runs for 1 second. Occasionally this is not long
enough and the perf report has no samples for symbol noploop.
Fix this and enlarge the runtime for the perf work load
to 3 seconds. This ensures the symbol noploop is always
present. Since only s390 is affected, make this loop
architecture dependend.
Output before:
Inject -b build-ids test
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.195 MB - ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.277 MB - ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.195 MB - ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.160 MB
/tmp/perf.data.ELzRdq (4031 samples) ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.195 MB - ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.195 MB - ]
Inject -b build-ids test [Success]
Inject --buildid-all build-ids test
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.195 MB - ]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.014 MB - ]
Inject --buildid-all build-ids test [Failed - cannot find
noploop function in pipe #2]
Output after:
Successful execution for over 10 times in a loop.
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Reviewed-by: Ian Rogers <irogers@google.com>
Suggested-by: Namhyung Kim <namhyung@kernel.org>
Cc: agordeev@linux.ibm.com
Cc: gor@linux.ibm.com
Cc: hca@linux.ibm.com
Link: https://lore.kernel.org/r/20241018081732.1391060-1-tmricht@linux.ibm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Like in the metricgroup tests, it should check the permission first and
then skip relevant failures accordingly.
Also it needs to try again with the system wide flag properly. On the
second round, check if the result has the metric name because other
failure cases are checked in the first round already.
Reviewed-by: Ian Rogers <irogers@google.com>
Link: https://lore.kernel.org/r/20241018204306.741972-1-namhyung@kernel.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
On my system, perf list is very slow to print the whole events. I think
there's a performance issue in SDT and uprobes event listing. I noticed
this issue while running perf test on x86 but it takes long to check
some CoreSight event which should be skipped quickly.
Anyway, some test uses perf list to check whether the required event is
available before running the test. The perf list command can take an
argument to specify event class or (glob) pattern. But glob pattern is
only to suppress output for unmatched ones after checking all events.
In this case, specifying event class is better to reduce the number of
events it checks and to avoid buggy subsystems entirely.
No functional changes intended.
Reviewed-by: James Clark <james.clark@linaro.org>
Reviewed-by: Ian Rogers <irogers@google.com>
Cc: German Gomez <german.gomez@arm.com>
Cc: Carsten Haitzler <carsten.haitzler@arm.com>
Cc: Leo Yan <leo.yan@arm.com>
Link: https://lore.kernel.org/r/20241016065654.269994-1-namhyung@kernel.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Command perf test 86 fails on s390:
# perf test -F 86
ping 868299 [007] 28248.013596: probe_libc:inet_pton_1: (3ff95948020)
3ff95948020 inet_pton+0x0 (inlined)
3ff9595e6e7 text_to_binary_address+0x1007 (inlined)
3ff9595e6e7 gaih_inet+0x1007 (inlined)
FAIL: expected backtrace entry \
"main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$"
got "3ff9595e6e7 gaih_inet+0x1007 (inlined)"
86: probe libc's inet_pton & backtrace it with ping : FAILED!
#
The root cause is a new stack layout, two functions have been added
as seen below.
# perf script | tac | grep -m1 '^ping' -B9 | tac
ping 866856 [007] 25979.494921: probe_libc:inet_pton: (3ff8ec48020)
3ff8ec48020 inet_pton+0x0 (inlined)
new --> 3ff8ec5e6e7 text_to_binary_address+0x1007 (inlined)
new --> 3ff8ec5e6e7 gaih_inet+0x1007 (inlined)
3ff8ec5e6e7 getaddrinfo+0x1007 (/usr/lib64/libc.so.6)
2aa3fe04bf5 main+0xff5 (/usr/bin/ping)
3ff8eb34a5b __libc_start_call_main+0x8b (/usr/lib64/libc.so.6)
3ff8eb34b5d __libc_start_main@GLIBC_2.2+0xad (inlined)
2aa3fe06a1f [unknown] (/usr/bin/ping)
#
The new functions in the call chain are:
- text_to_binary_address()
- gaih_inet().
Both functions are inlined and do not show up in the output
of the nm command:
# nm -a /usr/lib64/libc.so.6 | \
grep -E '(text_to_binary_address|gaih_inet)$'
#
There is no possibility to add these 2 functions depending on their
existance in the C library.
Add text_to_binary_address() and gaih_inet() to the list of
expected functions in an compatible way and extend the regular
expression. On s390 the backtrace can now be
Before After
probe_libc:inet_pton probe_libc:inet_pton
inet_pton inet_pton
getaddrinfo getaddrinfo | text_to_binary_address
main main | gaih_inet
Output after:
# perf test -F 86
86: probe libc's inet_pton & backtrace it with ping : Ok
#
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Cc: agordeev@linux.ibm.com
Cc: gor@linux.ibm.com
Cc: hca@linux.ibm.com
Cc: sumanthk@linux.ibm.com
Link: https://lore.kernel.org/r/20241001124224.3370306-1-tmricht@linux.ibm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
perf test 70 takes a long time. One culprit is the output of command
perf annotate. Per default enabled are
- demangle symbol names
- interleave source code with assembly code.
Disable demangle of symbols and abort the annotation
after the first 250 lines.
This speeds up the test case considerable, for example
on s390:
Output before:
# time perf test 70
70: perf annotate basic tests : Ok
.....
real 2m7.467s
user 1m26.869s
sys 0m34.086s
#
Output after:
# time perf test 70
70: perf annotate basic tests : Ok
real 0m3.341s
user 0m1.606s
sys 0m0.362s
#
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: James Clark <james.clark@linaro.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: sumanthk@linux.ibm.com
Link: https://lore.kernel.org/r/20240917085706.249691-1-tmricht@linux.ibm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
$ sudo ./perf test filtering -vv
96: perf record sample filtering (by BPF) tests:
--- start ---
test child forked, pid 2966908
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
---- end(0) ----
96: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://lore.kernel.org/r/20240826221045.1202305-5-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
As a form of validation, it is a common practice to check the outputs
of commands whether they contain expected patterns or match a certain
regular expression.
This output checking helper is designed to allow checking stderr output
of perf commands for unexpected messages, while ignoring messages that
are known to be harmless, e.g.:
"Lowering default frequency rate to \d+\."
"\d+ out of order events recorded."
etc.
Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
Cc: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20240702110849.31904-10-vmolnaro@redhat.com
Signed-off-by: Michael Petlan <mpetlan@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
