1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
|
CORE ANALYSIS SUITE
The core analysis suite is a self-contained tool that can be used to
investigate either live systems, kernel core dumps created from the
netdump and diskdump packages offered by Red Hat, the LKCD kernel patch
or the mcore kernel patch available from Mission Critical Linux.
o The tool is loosely based on the SVR4 crash command, but has been
completely integrated with gdb in order to be able to display
formatted kernel data structures, disassemble source code, etc.
o The current set of available commands consist of common kernel core
analysis tools such as a context-specific stack traces, source code
disassembly, kernel variable displays, memory display, dumps of
linked-lists, etc. In addition, any gdb command may be entered,
which in turn will be passed onto the gdb module for execution.
o There are several commands that delve deeper into specific kernel
subsystems, which also serve as templates for kernel developers
to create new commands for analysis of a specific area of interest.
Adding a new command is a simple affair, and a quick recompile
adds it to the command menu.
o The intent is to make the tool independent of Linux version dependencies,
building in recognition of major kernel code changes so as to adapt to
new kernel versions, while maintaining backwards compatibility.
A whitepaper with complete documentation concerning the use of this utility
can be found here:
http://people.redhat.com/anderson/crash_whitepaper
These are the current prerequisites:
o At this point, x86, ia64, x86_64, ppc64, alpha, s390 and s390x-based
kernels are supported. Other architectures may be addressed in the future.
o One size fits all -- the utility can be run on any Linux kernel version
from 2.2.5-15 through 2.6.*.
o In order to contain debugging data, the top-level kernel Makefile's CFLAGS
definition must contain the -g flag. If not, the kernel must be rebuilt.
For 2.2 kernels that are not built with -g, change the following line:
CFLAGS = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
to:
CFLAGS = -g -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
For 2.4 kernels that are not built with -g, change the following line:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing
to:
CFLAGS := -g $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing
For 2.6 kernels that are not built with -g, change the following line:
CFLAGS := -Wall -Wstrict-prototypes -Wno-trigraphs \
to:
CFLAGS := -g -Wall -Wstrict-prototypes -Wno-trigraphs \
After the kernel is re-compiled, the uncompressed "vmlinux" kernel
that is created in the top-level kernel build directory must be saved.
To build this utility, simply uncompress the tar file, enter the crash-4.0-6.3
subdirectory, and type "make". The initial build will take several minutes
because the gdb module must be configured and and built. Alternatively, the
crash source RPM file may be installed and built, and the resultant crash
binary RPM file installed.
If the tool is run against a crash dumpfile, two arguments are required, the
uncompressed kernel name and the core dumpfile name.
If run on a live system, only the kernel name is required, because /dev/mem
will be used as the "dumpfile". If the kernel file is stored in /boot, /,
/boot/efi, or in any /usr/src subdirectory, then no command line arguments
are required -- the first kernel found that matches /proc/version will be
used as the namelist.
For example, if the command name "crash" is kept intact in the Makefile,
invoking it on a live system would look like this:
$ crash
crash 4.0-6.3
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"..
KERNEL: /boot/vmlinux
DUMPFILE: /dev/mem
CPUS: 1
DATE: Tue Apr 29 13:39:16 2008
UPTIME: 10 days, 22:55:18
LOAD AVERAGE: 0.08, 0.03, 0.01
TASKS: 42
NODENAME: ha2.mclinux.com
RELEASE: 2.4.0-test10
VERSION: #11 SMP Thu Nov 4 15:09:25 EST 2000
MACHINE: i686 (447 MHz)
MEMORY: 128 MB
PID: 3621
COMMAND: "crash"
TASK: c463c000
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> help
* files mod runq union
alias foreach mount search vm
ascii fuser net set vtop
bt gdb p sig waitq
btop help ps struct whatis
dev irq pte swap wr
dis kmem ptob sym q
eval list ptov sys
exit log rd task
extend mach repeat timer
crash version: 4.0-6.3 gdb version: 6.1
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash>
When run on a dumpfile, both the kernel namelist and dumpfile must be
entered on the command line. For example, when run on a core dump created
by the Red Hat netdump or diskdump facilities:
$ crash vmlinux vmcore
crash 4.0-6.3
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux
DUMPFILE: vmcore
CPUS: 4
DATE: Tue Mar 2 13:57:09 2004
UPTIME: 00:02:40
LOAD AVERAGE: 2.24, 0.96, 0.37
TASKS: 70
NODENAME: pro1.lab.boston.redhat.com
RELEASE: 2.6.3-2.1.214.11smp
VERSION: #1 SMP Tue Mar 2 10:58:27 EST 2004
MACHINE: i686 (2785 Mhz)
MEMORY: 512 MB
PANIC: "Oops: 0002 [#1]" (check log for details)
PID: 0
COMMAND: "swapper"
TASK: 22fa200 (1 of 4) [THREAD_INFO: 2356000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
When run on a core dump created by the MCLX mcore patch:
$ crash vmlinux.17 lcore.cr.17
crash 4.0-6.3
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux.17
DUMPFILE: lcore.cr.17
CPUS: 1
DATE: Wed Nov 10 19:54:47 1999
UPTIME: 00:02:02
LOAD AVERAGE: 0.82, 0.34, 0.12
TASKS: 43
NODENAME: ha3.mclinux.com
RELEASE: 2.2.5-15
VERSION: #39 Wed Nov 10 17:43:16 CST 1999
MACHINE: i686 (501 MHz)
MEMORY: 96 MB
PANIC: "tulip_interrupt"
PID: 286
COMMAND: "in.rlogind"
TASK: c0b3a000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
When run on an LKCD core dump, the command line would be like so:
$ crash vmlinux vmdump.0
The tool's environment is context-specific. On a live system, the default
context is the command itself; on a dump the default context will be the
task that panicked. The most commonly-used commands are:
set - set a new task context by pid, task address, or cpu.
bt - backtrace of the current context, or as specified with arguments.
p - print the contents of a kernel variable.
rd - read memory, which may be either kernel virtual, user virtual, or
physical.
ps - simple process listing.
log - dump the kernel log_buf.
struct - print the contents of a structure at a specified address.
foreach - execute a command on all tasks, or those specified, in the system.
Detailed help concerning the use of each of the commands in the menu above
may be displayed by entering "help command", where "command" is one of those
listed above. Rather than getting bogged down in details here, simply
run the help command on each of the commands above. Note that many commands
have multiple options so as to avoid the proliferation of command names.
Command output may be piped to external commands or redirected to files.
Enter "help output" for details.
The command line history mechanism allows for command-line recall and
command-line editing. Input files containing a set of crash commands may
be substituted for command-line input. Enter "help input" for details.
Note that a .crashrc file (or .<your-command-name>rc if the name has been
changed), may contain any number of "set" or "alias" commands -- see the
help pages on those two commands for details.
Lastly, if a command is entered that is not recognized, it is checked
against the kernel's list of variables, structure, union or typedef names,
and if found, the command is passed to "p", "struct", "union" or "whatis".
That being the case, as long as a kernel variable/structure/union name is
different than any of the current commands.
(1) A kernel variable can be dumped by simply entering its name:
crash> init_mm
init_mm = $2 = {
mmap = 0xc022d540,
mmap_avl = 0x0,
mmap_cache = 0x0,
pgd = 0xc0101000,
count = {
counter = 0x6
},
map_count = 0x1,
mmap_sem = {
count = {
counter = 0x1
},
waking = 0x0,
wait = 0x0
},
context = 0x0,
start_code = 0xc0000000,
end_code = 0xc022b4c8,
end_data = c0250388,
...
(2) A structure or can be dumped simply by entering its name and address:
crash> vm_area_struct c5ba3910
struct vm_area_struct {
vm_mm = 0xc3ae3210,
vm_start = 0x821b000,
vm_end = 0x8692000,
vm_next = 0xc5ba3890,
vm_page_prot = {
pgprot = 0x25
},
vm_flags = 0x77,
vm_avl_height = 0x4,
vm_avl_left = 0xc0499540,
vm_avl_right = 0xc0499f40,
vm_next_share = 0xc04993c0,
vm_pprev_share = 0xc0499060,
vm_ops = 0x0,
vm_offset = 0x0,
vm_file = 0x0,
vm_pte = 0x0
}
The crash utility has been designed to facilitate the task of adding new
commands. New commands may be permanently compiled into the crash executable,
or dynamically added during runtime using shared object files.
To permanently add a new command to the crash executable's menu:
1. For a command named "xxx", put a reference to cmd_xxx() in defs.h.
2. Add cmd_xxx into the base_command_table[] array in global_data.c.
3. Write cmd_xxx(), putting it in one of the appropriate files. Look at
the other commands for guidance on getting symbolic data, reading
memory, displaying data, etc...
4. Recompile and run.
Note that while the initial compile of crash, which configures and compiles
the gdb module, takes several minutes, subsequent re-compiles to do such
things as add new commands or fix bugs just takes a few seconds.
Alternatively, you can create shared object library files consisting of
crash command extensions, that can be dynamically linked into the crash
executable during runtime or during initialization. This will allow the
the same shared object to be used with subsequent crash releases without
having to re-merge the command's code into each new set of crash sources.
The dynamically linked-in commands will automatically show up in the crash
help menu. For details, enter "help extend" during runtime, or enter
"crash -h extend" from the shell command line.
|
