1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
Description:
The cycle_thief.stp script instruments the scheduler and IRQ handler
to determine which processes and interrupts are competing with the
specified task for the cpu cycles. This script uses the '-c' or '-x'
options to focus on a specific task. The script output the number of
times the task migrates between processors, histograms showing the
length of time on and off processor, lists of processes running while
the task is off the processor, and the interrupts that occurred while
the task was running.
$ stap cycle_thief.stp
task 0 migrated: 1
task 0 on processor (us):
value |-------------------------------------------------- count
2 | 0
4 | 0
8 |@ 1
16 | 0
32 | 0
64 | 0
128 |@@@@@@@@ 8
256 |@ 1
512 | 0
1024 |@ 1
2048 |@@@@@@@@ 8
4096 |@@@@@@@@@@@ 11
8192 |@@ 2
16384 |@@ 2
32768 |@@@@@@@@@@@@ 12
65536 |@@@@@ 5
131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 30
262144 | 0
524288 | 0
task 0 off processor (us)
value |-------------------------------------------------- count
1 | 0
2 | 0
4 |@@@@ 4
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 27
16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 34
32 |@@@@ 4
64 |@@@@@ 5
128 |@@@@@ 5
256 | 0
512 |@ 1
1024 | 0
2048 | 0
other pids taking processor from task 0
PID count command
25145 33 stapio
19189 17 kworker/0:3
7 13 rcu_sched
22717 9 kworker/1:1
380 8 jbd2/dm-0-8
9 8 rcuos/1
570 6 kworker/0:1H
8 3 rcuos/0
478 3 systemd-journal
24747 3 kworker/0:1
491 2 systemd-udevd
19120 2 sshd
25142 2 stapio
615 1 alsactl
625 1 irqbalance
25144 1 stapio
31816 1 kworker/u4:1
25141 1 stap
724 1 NetworkManager
25143 1 systemd-udevd
irq taking processor from task 0
irq count min(us) avg(us) max(us)
44 9 3 8 19
The output from running the script for ~10 seconds.
The histograms provide a visual of the time a process spent on, or off. The
sampled processes are listed below the histograms to give a sense of what
was running at the time.
As mentioned in the description, the script can be focused on a certain process
with -c/-x.
$ stap cycle_thief.stp -T 10
task 29018 migrated: 2
task 29018 on processor (us):
value |-------------------------------------------------- count
2 | 0
4 | 0
8 |@ 1
16 | 0
32 |@ 1
64 |@ 1
128 | 0
256 |@ 1
512 | 0
1024 | 0
task 29018 off processor (us)
value |-------------------------------------------------- count
2 | 0
4 | 0
8 |@ 1
16 |@ 1
32 | 0
64 | 0
~
2097152 | 0
4194304 | 0
8388608 |@ 1
16777216 | 0
33554432 | 0
other pids taking processor from task 29018
PID count command
0 41 swapper/0
19189 24 kworker/0:3
7 12 rcu_sched
8 8 rcuos/0
29002 5 kworker/0:0
380 4 jbd2/dm-0-8
570 3 kworker/0:1H
6 3 kworker/u4:0
491 2 systemd-udevd
29016 2 stapio
29017 1 systemd-udevd
478 1 systemd-journal
625 1 irqbalance
14 1 migration/1
irq taking processor from task 29018
irq count min(us) avg(us) max(us)
|
