1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
|
TRACE-CMD-RECORD(1)
===================
NAME
----
trace-cmd-record - record a trace from the Ftrace Linux internal tracer
SYNOPSIS
--------
*trace-cmd record* ['OPTIONS'] ['command']
DESCRIPTION
-----------
The trace-cmd(1) record command will set up the Ftrace Linux kernel tracer to
record the specified plugins or events that happen while the 'command'
executes. If no command is given, then it will record until the user hits
Ctrl-C.
The record command of trace-cmd will set up the Ftrace tracer to start tracing
the various events or plugins that are given on the command line. It will then
create a number of tracing processes (one per CPU) that will start recording
from the kernel ring buffer straight into temporary files. When the command is
complete (or Ctrl-C is hit) all the files will be combined into a trace.dat
file that can later be read (see trace-cmd-report(1)).
OPTIONS
-------
*-p* 'plugin'::
Specify a trace plugin. Plugins are special Ftrace tracers that usually do
more than just trace an event. Common plugins are *function*,
*function_graph*, *preemptirqsoff*, *irqsoff*, *preemptoff*, and *wakeup*.
A plugin must be supported by the running kernel. To see a list of
available plugins, see trace-cmd-list(1).
*-e* 'event'::
Specify an event to trace. Various static trace points have been added to
the Linux kernel. They are grouped by subsystem where you can enable all
events of a given subsystem or specify specific events to be enabled. The
'event' is of the format "subsystem:event-name". You can also just specify
the subsystem without the ':event-name' or the event-name without the
"subsystem:". Using "-e sched_switch" will enable the "sched_switch" event
where as, "-e sched" will enable all events under the "sched" subsystem.
The 'event' can also contain glob expressions. That is, "*stat*" will
select all events (or subsystems) that have the characters "stat" in their
names.
The keyword 'all' can be used to enable all events.
*-a*::
Every event that is being recorded has its output format file saved
in the output file to be able to display it later. But if other
events are enabled in the trace without trace-cmd's knowledge, the
formats of those events will not be recorded and trace-cmd report will
not be able to display them. If this is the case, then specify the
*-a* option and the format for all events in the system will be saved.
*-T*::
Enable a stacktrace on each event. For example:
<idle>-0 [003] 58549.289091: sched_switch: kworker/0:1:0 [120] R ==> trace-cmd:2603 [120]
<idle>-0 [003] 58549.289092: kernel_stack: <stack trace>
=> schedule (ffffffff814b260e)
=> cpu_idle (ffffffff8100a38c)
=> start_secondary (ffffffff814ab828)
*--func-stack*::
Enable a stack trace on all functions. Note this is only applicable
for the "function" plugin tracer, and will only take effect if the
-l option is used and succeeds in limiting functions. If the function
tracer is not filtered, and the stack trace is enabled, you can live
lock the machine.
*-f* 'filter'::
Specify a filter for the previous event. This must come after a *-e*. This
will filter what events get recorded based on the content of the event.
Filtering is passed to the kernel directly so what filtering is allowed
may depend on what version of the kernel you have. Basically, it will
let you use C notation to check if an event should be processed or not.
----------------------------------------
==, >=, <=, >, <, &, |, && and ||
----------------------------------------
The above are usually safe to use to compare fields.
*-R* 'trigger'::
Specify a trigger for the previous event. This must come after a *-e*.
This will add a given trigger to the given event. To only enable the trigger
and not the event itself, then place the event after the *-v* option.
See Documentation/trace/events.txt in the Linux kernel source for more
information on triggers.
*-v*::
This will cause all events specified after it on the command line to not
be traced. This is useful for selecting a subsystem to be traced but to
leave out various events. For Example: "-e sched -v -e "\*stat\*"" will
enable all events in the sched subsystem except those that have "stat" in
their names.
Note: the *-v* option was taken from the way grep(1) inverts the following
matches.
*-F*::
This will filter only the executable that is given on the command line. If
no command is given, then it will filter itself (pretty pointless).
Using *-F* will let you trace only events that are caused by the given
command.
*-P* 'pid'::
Similar to *-F* but lets you specify a process ID to trace.
*-c*::
Used with either *-F* to trace the process' children too.
*-C* 'clock'::
Set the trace clock to "clock".
Use trace-cmd(1) list -C to see what clocks are available.
*-o* 'output-file'::
By default, trace-cmd report will create a 'trace.dat' file. You can
specify a different file to write to with the *-o* option.
*-l* 'function-name'::
This will limit the 'function' and 'function_graph' tracers to only trace
the given function name. More than one *-l* may be specified on the
command line to trace more than one function. The limited use of glob
expressions are also allowed. These are 'match\*' to only filter functions
that start with 'match'. '\*match' to only filter functions that end with
'match'. '\*match\*' to only filter on functions that contain 'match'.
*-g* 'function-name'::
This option is for the function_graph plugin. It will graph the given
function. That is, it will only trace the function and all functions that
it calls. You can have more than one *-g* on the command line.
*-n* 'function-name'::
This has the opposite effect of *-l*. The function given with the *-n*
option will not be traced. This takes precedence, that is, if you include
the same function for both *-n* and *-l*, it will not be traced.
*-d*::
Some tracer plugins enable the function tracer by default. Like the
latency tracers. This option prevents the function tracer from being
enabled at start up.
*-D*::
The option *-d* will try to use the function-trace option to disable the
function tracer (if available), otherwise it defaults to the proc file:
/proc/sys/kernel/ftrace_enabled, but will not touch it if the function-trace
option is available. The *-D* option will disable both the ftrace_enabled
proc file as well as the function-trace option if it exists.
Note, this disable function tracing for all users, which includes users
outside of ftrace tracers (stack_tracer, perf, etc).
*-O* 'option'::
Ftrace has various options that can be enabled or disabled. This allows
you to set them. Appending the text 'no' to an option disables it.
For example: "-O nograph-time" will disable the "graph-time" Ftrace
option.
*-s* 'interval'::
The processes that trace-cmd creates to record from the ring buffer need
to wake up to do the recording. Setting the 'interval' to zero will cause
the processes to wakeup every time new data is written into the buffer.
But since Ftrace is recording kernel activity, the act of this processes
going back to sleep may cause new events into the ring buffer which will
wake the process back up. This will needlessly add extra data into the
ring buffer.
The 'interval' metric is microseconds. The default is set to 1000 (1 ms).
This is the time each recording process will sleep before waking up to
record any new data that was written to the ring buffer.
*-r* 'priority'::
The priority to run the capture threads at. In a busy system the trace
capturing threads may be staved and events can be lost. This increases
the priority of those threads to the real time (FIFO) priority.
But use this option with care, it can also change the behaviour of
the system being traced.
*-b* 'size'::
This sets the ring buffer size to 'size' kilobytes. Because the Ftrace
ring buffer is per CPU, this size is the size of each per CPU ring buffer
inside the kernel. Using "-b 10000" on a machine with 4 CPUs will make
Ftrace have a total buffer size of 40 Megs.
*-B* 'buffer-name'::
If the kernel supports multiple buffers, this will add a buffer with
the given name. If the buffer name already exists, that buffer is just
reset and will not be deleted at the end of record execution. If the
buffer is created, it will be removed at the end of execution (unless
the *-k* is set, or 'start' command was used).
After a buffer name is stated, all events added after that will be
associated with that buffer. If no buffer is specified, or an event
is specified before a buffer name, it will be associated with the
main (toplevel) buffer.
trace-cmd record -e sched -B block -e block -B time -e timer sleep 1
The above is will enable all sched events in the main buffer. It will
then create a 'block' buffer instance and enable all block events within
that buffer. A 'time' buffer instance is created and all timer events
will be enabled for that event.
*-m* 'size'::
The max size in kilobytes that a per cpu buffer should be. Note, due
to rounding to page size, the number may not be totally correct.
Also, this is performed by switching between two buffers that are half
the given size thus the output may not be of the given size even if
much more was written.
Use this to prevent running out of diskspace for long runs.
*-M* 'cpumask'::
Set the cpumask for to trace. It only affects the last buffer instance
given. If supplied before any buffer instance, then it affects the
main buffer. The value supplied must be a hex number.
trace-cmd record -p function -M c -B events13 -e all -M 5
If the -M is left out, then the mask stays the same. To enable all
CPUs, pass in a value of '-1'.
*-k*::
By default, when trace-cmd is finished tracing, it will reset the buffers
and disable all the tracing that it enabled. This option keeps trace-cmd
from disabling the tracer and reseting the buffer. This option is useful for
debugging trace-cmd.
Note: usually trace-cmd will set the "tracing_on" file back to what it
was before it was called. This option will leave that file set to zero.
*-i*::
By default, if an event is listed that trace-cmd does not find, it
will exit with an error. This option will just ignore events that are
listed on the command line but are not found on the system.
*-N* 'host:port'::
If another machine is running "trace-cmd listen", this option is used to
have the data sent to that machine with UDP packets. Instead of writing
to an output file, the data is sent off to a remote box. This is ideal for
embedded machines with little storage, or having a single machine that
will keep all the data in a single repository.
Note: This option is not supported with latency tracer plugins:
wakeup, wakeup_rt, irqsoff, preemptoff and preemptirqsoff
*-t*::
This option is used with *-N*, when there's a need to send the live data
with TCP packets instead of UDP. Although TCP is not nearly as fast as
sending the UDP packets, but it may be needed if the network is not that
reliable, the amount of data is not that intensive, and a guarantee is
needed that all traced information is transfered successfully.
*--date*::
With the *--date* option, "trace-cmd" will write timestamps into the
trace buffer after it has finished recording. It will then map the
timestamp to gettimeofday which will allow wall time output from the
timestamps reading the created 'trace.dat' file.
*--profile*::
With the *--profile* option, "trace-cmd" will enable tracing that can
be used with trace-cmd-report(1) --profile option. If a tracer *-p* is
not set, and function graph depth is supported by the kernel, then
the function_graph tracer will be enabled with a depth of one (only
show where userspace enters into the kernel). It will also enable
various tracepoints with stack tracing such that the report can show
where tasks have been blocked for the longest time.
See trace-cmd-profile(1) for more details and examples.
*-H* 'event-hooks'::
Add custom event matching to connect any two events together. When not
used with *--profile*, it will save the parameter and this will be
used by trace-cmd report --profile, too. That is:
trace-cmd record -H hrtimer_expire_entry,hrtimer/hrtimer_expire_exit,hrtimer,sp
trace-cmd report --profile
Will profile hrtimer_expire_entry and hrtimer_expire_ext times.
See trace-cmd-profile(1) for format.
*--stderr*::
Have output go to stderr instead of stdout, but the output of the command
executed will not be changed. This is useful if you want to monitor the
output of the command being executed, but not see the output from trace-cmd.
EXAMPLES
--------
The basic way to trace all events:
------------------------------
# trace-cmd record -e all ls > /dev/null
# trace-cmd report
trace-cmd-13541 [003] 106260.693809: filemap_fault: address=0x128122 offset=0xce
trace-cmd-13543 [001] 106260.693809: kmalloc: call_site=81128dd4 ptr=0xffff88003dd83800 bytes_req=768 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_ZERO
ls-13545 [002] 106260.693809: kfree: call_site=810a7abb ptr=0x0
ls-13545 [002] 106260.693818: sys_exit_write: 0x1
------------------------------
To use the function tracer with sched switch tracing:
------------------------------
# trace-cmd record -p function -e sched_switch ls > /dev/null
# trace-cmd report
ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair
ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120
trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault
trace-cmd-13586 [003] 106467.860314: function: up_read <-- do_page_fault
ls-13587 [002] 106467.860317: function: __phys_addr <-- schedule
trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault
ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to
trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault
------------------------------
Here is a nice way to find what interrupts have the highest latency:
------------------------------------------
# trace-cmd record -p function_graph -e irq_handler_entry -l do_IRQ sleep 10
# trace-cmd report
<idle>-0 [000] 157412.933969: funcgraph_entry: | do_IRQ() {
<idle>-0 [000] 157412.933974: irq_handler_entry: irq=48 name=eth0
<idle>-0 [000] 157412.934004: funcgraph_exit: + 36.358 us | }
<idle>-0 [000] 157413.895004: funcgraph_entry: | do_IRQ() {
<idle>-0 [000] 157413.895011: irq_handler_entry: irq=48 name=eth0
<idle>-0 [000] 157413.895026: funcgraph_exit: + 24.014 us | }
<idle>-0 [000] 157415.891762: funcgraph_entry: | do_IRQ() {
<idle>-0 [000] 157415.891769: irq_handler_entry: irq=48 name=eth0
<idle>-0 [000] 157415.891784: funcgraph_exit: + 22.928 us | }
<idle>-0 [000] 157415.934869: funcgraph_entry: | do_IRQ() {
<idle>-0 [000] 157415.934874: irq_handler_entry: irq=48 name=eth0
<idle>-0 [000] 157415.934906: funcgraph_exit: + 37.512 us | }
<idle>-0 [000] 157417.888373: funcgraph_entry: | do_IRQ() {
<idle>-0 [000] 157417.888381: irq_handler_entry: irq=48 name=eth0
<idle>-0 [000] 157417.888398: funcgraph_exit: + 25.943 us | }
------------------------------------------
An example of the profile:
------------------------------------------
# trace-cmd record --profile sleep 1
# trace-cmd report --profile --comm sleep
task: sleep-21611
Event: sched_switch:R (1) Total: 99442 Avg: 99442 Max: 99442 Min:99442
<stack> 1 total:99442 min:99442 max:99442 avg=99442
=> ftrace_raw_event_sched_switch (0xffffffff8105f812)
=> __schedule (0xffffffff8150810a)
=> preempt_schedule (0xffffffff8150842e)
=> ___preempt_schedule (0xffffffff81273354)
=> cpu_stop_queue_work (0xffffffff810b03c5)
=> stop_one_cpu (0xffffffff810b063b)
=> sched_exec (0xffffffff8106136d)
=> do_execve_common.isra.27 (0xffffffff81148c89)
=> do_execve (0xffffffff811490b0)
=> SyS_execve (0xffffffff811492c4)
=> return_to_handler (0xffffffff8150e3c8)
=> stub_execve (0xffffffff8150c699)
Event: sched_switch:S (1) Total: 1000506680 Avg: 1000506680 Max: 1000506680 Min:1000506680
<stack> 1 total:1000506680 min:1000506680 max:1000506680 avg=1000506680
=> ftrace_raw_event_sched_switch (0xffffffff8105f812)
=> __schedule (0xffffffff8150810a)
=> schedule (0xffffffff815084b8)
=> do_nanosleep (0xffffffff8150b22c)
=> hrtimer_nanosleep (0xffffffff8108d647)
=> SyS_nanosleep (0xffffffff8108d72c)
=> return_to_handler (0xffffffff8150e3c8)
=> tracesys_phase2 (0xffffffff8150c304)
Event: sched_wakeup:21611 (1) Total: 30326 Avg: 30326 Max: 30326 Min:30326
<stack> 1 total:30326 min:30326 max:30326 avg=30326
=> ftrace_raw_event_sched_wakeup_template (0xffffffff8105f653)
=> ttwu_do_wakeup (0xffffffff810606eb)
=> ttwu_do_activate.constprop.124 (0xffffffff810607c8)
=> try_to_wake_up (0xffffffff8106340a)
------------------------------------------
SEE ALSO
--------
trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1),
trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1),
trace-cmd-list(1), trace-cmd-listen(1), trace-cmd-profile(1)
AUTHOR
------
Written by Steven Rostedt, <rostedt@goodmis.org>
RESOURCES
---------
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
COPYING
-------
Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
the terms of the GNU Public License (GPL).
|
