1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
|
Core dump analysis
Core dumps can be a valuable source of information in certain troubleshooting scenarios. Detailed analysis of core dumps requires knowledge of the programming language(s) the problematic application (and the libraries and plugins it depends on) is written in, as well as a development environment that matches the environment in which the core dump was taken in terms of architecture as well as application, library and plugin versions. The supportconfig tool (http://www.novell.com/communities/node/2332) provides a convenient way of capturing this environment information.
As core dump analysis requires specialist skills and can be a highly complex process, it should generally only be attempted when more straightforward troubleshooting steps (in particular, applying relevant patches) have not brought relief.
When do core dumps occur?
A process will crash if it receives a signal (which it is not ignoring, or which it cannot ignore) and for which it has not set up a signal handler routine. For example, most processes generally do not set up a signal handler to handle SIGSEGV and will crash when the kernel sends them this signal to indicate that they have tried to perform an incorrect memory access. When this situation occurs and certain environmental settings are right, the kernel will write a core dump file for the process.
Preparation step: Disable the limit for the maximum size of a core dump file
On Unix systems, limits can be set for various resources available to processes. These limits can be displayed through "ulimit -a". One limitable value is the size of core dumps. To ensure a complete core dump can be written, run "ulimit -c unlimited" prior to starting the application of which the core dump needs to be captured. Changing the limit does not affect running processes and only applies to processes started the current shell.
To change limits permanently, install the ulimit package, configure the limits in the/etc/sysconfig/ulimit configuration file and reboot the system for the changed limits to take effect.
Preparation step: Configure a fixed location for storing core dumps
By default, the kernel writes core dump files in the current working directory of the crashing process (if file system permissions and ACLs allow). This can make it difficult to locate core dump files. To store core dumps in a fixed directory, first create a suitable directory, say /var/local/dumps:
install -m 1777 -d /var/local/dumps
(1777 corresponds to world writable, deletable only by owner), next instruct the kernel to store dumps there:
echo"/var/local/dumps/core.%e.%p"> /proc/sys/kernel/core_pattern
The kernel will expand %e to the name of the crashing process and %p to the PID.
To additionally make this configuration change persistent over reboots, add a line
kernel.core_pattern=/var/local/dumps/core.%e.%p
to the /etc/sysctl.conf configuration file.
For additional details on the format of the core_pattern parameter, refer to /usr/src/linux/Documentation/sysctl/kernel.txt in the kernel sources.
Preparation step: Disable AppArmor
AppArmor application security is based on learning the normal/good behaviour of an application and then preventing the application from performing operations that do not fit the behaviour learnt. As writing a core dump is typically not part of an application's normal/good behaviour, AppArmor is likely to prevent a core file from being written.
Refer to the AppArmor documentation for details on how to disable AppArmor selectively, or execute
rcapparmor stop
to disable AppArmor completely (at the cost of loosing AppArmor's protection).
Preparation step: Enable core dumps for setuid and setgid processes
Unix systems provide a facility, the setuid and setgid bits, to execute processes under a different user id or group id than that of the user/group starting the process. Such a process may be dealing with data which should not be directly accessible to the invoking user/group and if a core dump were written for such a process, that data could be leaked. For this reason, the kernel does not write core dumps for setuid/setgid processes by default. Similarly, by default the kernel will not write core dumps for processes running as the root user when doing so could leak information. This default behaviour can be overridden through the kernel.suid_dumpable sysctl:
sysctl -w kernel.suid_dumpable=2
Refer to /usr/src/linux/Documentation/sysctl/fs.txt in the kernel sources for details on this sysctl.
Manually triggering a core dump
In some troubleshooting scenarios, it may be necessary to trigger the generation of a core dump manually. This can be done by sending a signal that generates a coredump to the process, for example SIGABRT. To send SIGABRT to process 1234, run
kill -ABRT 1234
or to send SIGABRT to all running firefox-bin processes, run
kill -ABRT $(pidof firefox-bin)
|
