1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
|
Demonstrations of opensnoop, the Linux eBPF/bcc version.
opensnoop traces the open() syscall system-wide, and prints various details.
Example output:
# ./opensnoop
PID COMM FD ERR PATH
17326 <...> 7 0 /sys/kernel/debug/tracing/trace_pipe
1576 snmpd 9 0 /proc/net/dev
1576 snmpd 11 0 /proc/net/if_inet6
1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms
1576 snmpd 11 0 /proc/sys/net/ipv6/conf/lo/forwarding
1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms
1576 snmpd 9 0 /proc/diskstats
1576 snmpd 9 0 /proc/stat
1576 snmpd 9 0 /proc/vmstat
1956 supervise 9 0 supervise/status.new
1956 supervise 9 0 supervise/status.new
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libtinfo.so.5
17358 run 3 0 /lib/x86_64-linux-gnu/libdl.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libc.so.6
17358 run -1 6 /dev/tty
17358 run 3 0 /proc/meminfo
17358 run 3 0 /etc/nsswitch.conf
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_compat.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libnsl.so.1
17358 run 3 0 /etc/ld.so.cache
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_nis.so.2
17358 run 3 0 /lib/x86_64-linux-gnu/libnss_files.so.2
17358 run 3 0 /etc/passwd
17358 run 3 0 ./run
^C
While tracing, the snmpd process opened various /proc files (reading metrics),
and a "run" process read various libraries and config files (looks like it
was starting up: a new process).
opensnoop can be useful for discovering configuration and log files, if used
during application startup.
The -p option can be used to filter on a PID, which is filtered in-kernel. Here
I've used it with -T to print timestamps:
./opensnoop -Tp 1956
TIME(s) PID COMM FD ERR PATH
0.000000000 1956 supervise 9 0 supervise/status.new
0.000289999 1956 supervise 9 0 supervise/status.new
1.023068000 1956 supervise 9 0 supervise/status.new
1.023381997 1956 supervise 9 0 supervise/status.new
2.046030000 1956 supervise 9 0 supervise/status.new
2.046363000 1956 supervise 9 0 supervise/status.new
3.068203997 1956 supervise 9 0 supervise/status.new
3.068544999 1956 supervise 9 0 supervise/status.new
This shows the supervise process is opening the status.new file twice every
second.
The -U option include UID on output:
# ./opensnoop -U
UID PID COMM FD ERR PATH
0 27063 vminfo 5 0 /var/run/utmp
103 628 dbus-daemon -1 2 /usr/local/share/dbus-1/system-services
103 628 dbus-daemon 18 0 /usr/share/dbus-1/system-services
103 628 dbus-daemon -1 2 /lib/dbus-1/system-services
The -u option filtering UID:
# ./opensnoop -Uu 1000
UID PID COMM FD ERR PATH
1000 30240 ls 3 0 /etc/ld.so.cache
1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libselinux.so.1
1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libc.so.6
1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libpcre.so.3
1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libdl.so.2
1000 30240 ls 3 0 /lib/x86_64-linux-gnu/libpthread.so.0
The -x option only prints failed opens:
# ./opensnoop -x
PID COMM FD ERR PATH
18372 run -1 6 /dev/tty
18373 run -1 6 /dev/tty
18373 multilog -1 13 lock
18372 multilog -1 13 lock
18384 df -1 2 /usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo
18384 df -1 2 /usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo
18384 df -1 2 /usr/share/locale/en_US/LC_MESSAGES/coreutils.mo
18384 df -1 2 /usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo
18384 df -1 2 /usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo
18384 df -1 2 /usr/share/locale/en/LC_MESSAGES/coreutils.mo
18385 run -1 6 /dev/tty
18386 run -1 6 /dev/tty
This caught a df command failing to open a coreutils.mo file, and trying from
different directories.
The ERR column is the system error number. Error number 2 is ENOENT: no such
file or directory.
A maximum tracing duration can be set with the -d option. For example, to trace
for 2 seconds:
# ./opensnoop -d 2
PID COMM FD ERR PATH
2191 indicator-multi 11 0 /sys/block
2191 indicator-multi 11 0 /sys/block
2191 indicator-multi 11 0 /sys/block
2191 indicator-multi 11 0 /sys/block
2191 indicator-multi 11 0 /sys/block
The -n option can be used to filter on process name using partial matches:
# ./opensnoop -n ed
PID COMM FD ERR PATH
2679 sed 3 0 /etc/ld.so.cache
2679 sed 3 0 /lib/x86_64-linux-gnu/libselinux.so.1
2679 sed 3 0 /lib/x86_64-linux-gnu/libc.so.6
2679 sed 3 0 /lib/x86_64-linux-gnu/libpcre.so.3
2679 sed 3 0 /lib/x86_64-linux-gnu/libdl.so.2
2679 sed 3 0 /lib/x86_64-linux-gnu/libpthread.so.0
2679 sed 3 0 /proc/filesystems
2679 sed 3 0 /usr/lib/locale/locale-archive
2679 sed -1 2
2679 sed 3 0 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
2679 sed 3 0 /dev/null
2680 sed 3 0 /etc/ld.so.cache
2680 sed 3 0 /lib/x86_64-linux-gnu/libselinux.so.1
2680 sed 3 0 /lib/x86_64-linux-gnu/libc.so.6
2680 sed 3 0 /lib/x86_64-linux-gnu/libpcre.so.3
2680 sed 3 0 /lib/x86_64-linux-gnu/libdl.so.2
2680 sed 3 0 /lib/x86_64-linux-gnu/libpthread.so.0
2680 sed 3 0 /proc/filesystems
2680 sed 3 0 /usr/lib/locale/locale-archive
2680 sed -1 2
^C
This caught the 'sed' command because it partially matches 'ed' that's passed
to the '-n' option.
The -e option prints out extra columns; for example, the following output
contains the flags passed to open(2), in octal:
# ./opensnoop -e
PID COMM FD ERR FLAGS PATH
28512 sshd 10 0 00101101 /proc/self/oom_score_adj
28512 sshd 3 0 02100000 /etc/ld.so.cache
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libwrap.so.0
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libaudit.so.1
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libpam.so.0
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libselinux.so.1
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libsystemd.so.0
28512 sshd 3 0 02100000 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
28512 sshd 3 0 02100000 /lib/x86_64-linux-gnu/libutil.so.1
The -f option filters based on flags to the open(2) call, for example:
# ./opensnoop -e -f O_WRONLY -f O_RDWR
PID COMM FD ERR FLAGS PATH
28084 clear_console 3 0 00100002 /dev/tty
28084 clear_console -1 13 00100002 /dev/tty0
28084 clear_console -1 13 00100001 /dev/tty0
28084 clear_console -1 13 00100002 /dev/console
28084 clear_console -1 13 00100001 /dev/console
28051 sshd 8 0 02100002 /var/run/utmp
28051 sshd 7 0 00100001 /var/log/wtmp
The --cgroupmap option filters based on a cgroup set. It is meant to be used
with an externally created map.
# ./opensnoop --cgroupmap /sys/fs/bpf/test01
For more details, see docs/special_filtering.md
USAGE message:
# ./opensnoop -h
usage: opensnoop.py [-h] [-T] [-U] [-x] [-p PID] [-t TID]
[--cgroupmap CGROUPMAP] [--mntnsmap MNTNSMAP] [-u UID]
[-d DURATION] [-n NAME] [-e] [-f FLAG_FILTER] [-F]
[-b BUFFER_PAGES]
Trace open() syscalls
optional arguments:
-h, --help show this help message and exit
-T, --timestamp include timestamp on output
-U, --print-uid print UID column
-x, --failed only show failed opens
-p PID, --pid PID trace this PID only
-t TID, --tid TID trace this TID only
--cgroupmap CGROUPMAP
trace cgroups in this BPF map only
--mntnsmap MNTNSMAP trace mount namespaces in this BPF map only
-u UID, --uid UID trace this UID only
-d DURATION, --duration DURATION
total duration of trace in seconds
-n NAME, --name NAME only print process names containing this name
-e, --extended_fields
show extended fields
-f FLAG_FILTER, --flag_filter FLAG_FILTER
filter on flags argument (e.g., O_WRONLY)
-F, --full-path show full path for an open file with relative path
-b BUFFER_PAGES, --buffer-pages BUFFER_PAGES
size of the perf ring buffer (must be a power of two
number of pages and defaults to 64)
examples:
./opensnoop # trace all open() syscalls
./opensnoop -T # include timestamps
./opensnoop -U # include UID
./opensnoop -x # only show failed opens
./opensnoop -p 181 # only trace PID 181
./opensnoop -t 123 # only trace TID 123
./opensnoop -u 1000 # only trace UID 1000
./opensnoop -d 10 # trace for 10 seconds only
./opensnoop -n main # only print process names containing "main"
./opensnoop -e # show extended fields
./opensnoop -f O_WRONLY -f O_RDWR # only print calls for writing
./opensnoop -F # show full path for an open file with relative path
./opensnoop --cgroupmap mappath # only trace cgroups in this BPF map
./opensnoop --mntnsmap mappath # only trace mount namespaces in the map
|
