1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
<title>Diagnosing Problems</title>
</head>
<body alink="#FF0000" vlink="#55188A" link="#0000EF" bgcolor="#FFFFFF" text="#000099">
<table border="0">
<tr align="left">
<td valign="top">
<table border="0">
<tr align="left"><td valign="top" >
<img src="http://user-mode-linux.sourceforge.net/uml-small.png" height="171" width="120"/>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/index.html">Site Home Page</a></font>
<br>
<font size="-1"><a href="http://uml.harlowhill.com">The UML Wiki</a></font>
<br>
<font size="-1"><a href="http://usermodelinux.org">UML Community Site</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/roadmap.html">The UML roadmap</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/uses.html">What it's good for</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/case-studies.html">Case Studies</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/kernel.html">Kernel Capabilities</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/dl-sf.html">Downloading it</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/run.html">Running it</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/compile.html">Compiling</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/install.html">Installation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/skas.html">Skas Mode</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/patches.html">Incremental Patches</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/tests.html">Test Suite</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/devanon.html">Host memory use</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/fs_making.html">Building filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/faq.html">Troubles</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/contrib.html">User Contributions</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/links.html">Related Links</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/todo.html">The ToDo list</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/projects.html">Projects</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/diary.html">Diary</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/thanks.html">Thanks</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/contacts.html">Contacts</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Tutorials<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/UserModeLinux-HOWTO.html">The HOWTO (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/UserModeLinux-HOWTO.txt">The HOWTO (text)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/hostfs.html">Host file access</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/input.html">Device inputs</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/shared_fs.html">Sharing filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/fs.html">Creating filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/resize.html">Resizing filesystems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/networking.html">Virtual Networking</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/mconsole.html">Management Console</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/debugging.html">Kernel Debugging</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/honeypots.html">UML Honeypots</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/gprof.html">gprof and gcov</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/xtut.html">Running X</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/trouble.html">Diagnosing problems</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/config.html">Configuration</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slack_readme.html">Installing Slackware</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/arch-port.html">Porting UML</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/iomem.html">IO memory emulation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/2G-2G.html">UML on 2G/2G hosts</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/lksct/index.html">Adding a UML system call</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/nesting.html">Running nested UMLs</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">How you can help<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-gen.html">Overview</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-doc.html">Documentation</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-userspace.html">Utilities</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/help-kernel-v1.html">Kernel bugs</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/projects.html">Kernel projects</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Screenshots<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/net.html">A virtual network</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/x.html">An X session</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Transcripts<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/login.html">A login session</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/debug-session.html">A debugging session</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slackinst.html">Slackware installation</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Reference<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/switches.html">Kernel switches</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slack_readme.html">Slackware README</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Papers<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2000/index.html">ALS 2000 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2000.tex">ALS 2000 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/als2000/slides.html">ALS 2000 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lca2001/lca.html">LCA 2001 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ols2001/index.html">OLS 2001 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/ols2001.tex">OLS 2001 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2001/index.html">ALS 2001 paper (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/als2001.tex">ALS 2001 paper (TeX)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ists2002/umlsec.htm">UML security (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lca2002/lca2002.htm">LCA 2002 (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/wvu2002/wvu2002.htm">WVU 2002 (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ists_rt/ists_rt.htm">Security Roundtable (html)</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/ols2002/ols2002.html">OLS 2002 slides</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/slides/lwe2005/LWE2005.html">LWE 2005 slides</a></font>
</td></tr>
<tr align="left"><td valign="top" bgcolor="#e0e0e0">Fun and Games<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/cgi-bin/hangman">Kernel Hangman</a></font>
<br>
<font size="-1"><a href="http://user-mode-linux.sourceforge.net/sdotm.html">Disaster of the Month</a></font>
</td></tr>
</table>
</td>
<td valign="top" align="left">
<center>
<h3>Diagnosing Problems</h3>
</center>
If you get UML to crash, hang, or otherwise misbehave, you should
report this on one of the project mailing lists, either the
developer list - user-mode-linux-devel at lists dot sourceforge dot
net (<a href="http://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel">subscription info</a>) or the user list -
user-mode-linux-user at lists dot sourceforge dot net
(<a href="http://lists.sourceforge.net/lists/listinfo/user-mode-linux-user">subscription info</a>). When you do, it is
likely that I will want more information. So, it would be helpful to
read the stuff below, do whatever is applicable in your case, and
report the results to the list.
<p>
For any diagnosis, you're going to need to build a debugging kernel.
The binaries from this site aren't debuggable. If you haven't done
this before, read about
<a href="compile.html">compiling</a> and
<a href="debugging.html">debugging</a> UML first.
<a name="Case 1 : Normal kernel panics"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Case 1 : Normal kernel panics</font>
</b>
</td>
</tr>
</table>
<blockquote head="Case 1 : Normal kernel panics">
The most common case is for a normal thread to panic. To debug this,
you will need to run it under the debugger (add 'debug' to the command
line). An xterm will start up with gdb running inside it. Continue
it when it stops in start_kernel and make it crash. Now ^C gdb and
'bt'. I'm going to want to see the resulting stack trace.
<p>
If the panic was a "Kernel mode fault", then there will be a segv
frame on the stack and I'm going to want some more information. The
stack might look something like this:<tt>
<pre>
<font size="-1">
<font color="#000000" where="UML gdb">(UML gdb) backtrace</font>
#0 0x1009bf76 in __sigprocmask (how=1, set=0x5f347940, oset=0x0)
at ../sysdeps/unix/sysv/linux/sigprocmask.c:49
#1 0x10091411 in change_sig (signal=10, on=1) at process.c:218
#2 0x10094785 in timer_handler (sig=26) at time_kern.c:32
#3 0x1009bf38 in __restore ()
at ../sysdeps/unix/sysv/linux/i386/sigaction.c:125
#4 0x1009534c in segv (address=8, ip=268849158, is_write=2, is_user=0)
at trap_kern.c:66
#5 0x10095c04 in segv_handler (sig=11) at trap_user.c:285
#6 0x1009bf38 in __restore ()
</font>
</pre>
</tt>
I'm going to want to see the symbol and line information for the value
of ip in the segv frame. In this case, you would do the following:
<blockquote>
<tt>
<font color="#000000" where="UML gdb">(UML gdb) i sym 268849158</font>
</tt>
</blockquote>
and
<blockquote>
<tt>
<font color="#000000" where="UML gdb">(UML gdb) i line *268849158</font>
</tt>
</blockquote>
The reason for this is the __restore frame right above the
segv_handler frame is hiding the frame that actually segfaulted. So,
I have to get that information from the faulting ip.
</blockquote>
<a name="Case 2 : Tracing thread panics"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Case 2 : Tracing thread panics</font>
</b>
</td>
</tr>
</table>
<blockquote head="Case 2 : Tracing thread panics">
The less common and more painful case is when the tracing thread
panics. In this case, the kernel debugger will be useless because it
needs a healthy tracing thread in order to work. The first thing to
do is get a backtrace from the tracing thread. This is done by
figuring out what its pid is, firing up gdb, and attaching it to that
pid. You can figure out the tracing thread pid by looking at the
first line of the console output, which will look like this:
<tt>
<pre>
<font size="-1">tracing thread pid = 15851</font>
</pre>
</tt>
or by running ps on the host and finding the line that looks like
this:
<tt>
<pre>
<font size="-1">
jdike 15851 4.5 0.4 132568 1104 pts/0 S 21:34 0:05 ./linux [(tracing thread)]
</font>
</pre>
</tt>
If the panic was 'segfault in signals', then follow the instructions
above for collecting information about the location of the seg fault.
<p>
If the tracing thread flaked out all by itself, then send that
backtrace in and wait for our crack debugging team to fix the
problem.
</blockquote>
<a name="Case 3 : Tracing thread panics caused by other threads"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Case 3 : Tracing thread panics caused by other threads</font>
</b>
</td>
</tr>
</table>
<blockquote head="Case 3 : Tracing thread panics caused by other threads">
However, there are cases where the misbehavior of another
thread caused the problem. The most common panic of this type is:
<tt>
<pre>
<font size="-1">
wait_for_stop failed to wait for <i>pid</i> to stop with <i>signal number</i>
</font>
</pre>
</tt>
In this case, you'll need to get a backtrace from the process
mentioned in the panic, which is complicated by the fact that the
kernel debugger is defunct and without some fancy footwork, another
gdb can't attach to it. So, this is how the fancy footwork goes:<br>
In a shell:
<blockquote>
<tt>
<font color="#000000" where="host">host% kill -STOP <i>pid</i>
</font>
</tt>
</blockquote>
Run gdb on the tracing thread as described in case 2 and do:
<blockquote>
<tt>
<font color="#000000" where="host gdb">(host gdb) call detach(<i>pid</i>)</font>
</tt>
</blockquote>
If you get a segfault, do it again. It always works the second
time.<br>
Detach from the tracing thread and attach to that other thread:
<blockquote>
<tt>
<font color="#000000" where="host gdb">(host gdb) detach</font>
</tt>
</blockquote>
<blockquote>
<tt>
<font color="#000000" where="host gdb">(host gdb) attach <i>pid</i>
</font>
</tt>
</blockquote>
If gdb hangs when attaching to that process, go back to a shell and
do:
<blockquote>
<tt>
<font color="#000000" where="host">host%
kill -CONT <i>pid</i>
</font>
</tt>
</blockquote>
And then get the backtrace:
<blockquote>
<tt>
<font color="#000000" where="host gdb">(host gdb) backtrace</font>
</tt>
</blockquote>
</blockquote>
<a name="Case 4 : Hangs"/><table width="100%" bgcolor="#e0e0e0">
<tr>
<td>
<b>
<font color="black">Case 4 : Hangs</font>
</b>
</td>
</tr>
</table>
<blockquote head="Case 4 : Hangs">
Hangs seem to be fairly rare, but they sometimes happen. When a hang
happens, we need a backtrace from the offending process. Run the
kernel debugger as described in case 1 and get a backtrace. If the
current process is not the idle thread, then send in the backtrace.
You can tell that it's the idle thread if the stack looks like this:
<tt>
<pre>
<font size="-1">
#0 0x100b1401 in __libc_nanosleep ()
#1 0x100a2885 in idle_sleep (secs=10) at time.c:122
#2 0x100a546f in do_idle () at process_kern.c:445
#3 0x100a5508 in cpu_idle () at process_kern.c:471
#4 0x100ec18f in start_kernel () at init/main.c:592
#5 0x100a3e10 in start_kernel_proc (unused=0x0) at um_arch.c:71
#6 0x100a383f in signal_tramp (arg=0x100a3dd8) at trap_user.c:50
</font>
</pre>
</tt>
If this is the case, then some other process is at fault, and went to
sleep when it shouldn't have. Run ps on the host and figure out which
process should not have gone to sleep and stayed asleep. Then attach
to it with gdb and get a backtrace as described in case 3.
</blockquote>
</td>
</tr>
</table>
<center>
<font size="-1">Hosted at </font>
<a href="http://sourceforge.net">
<img src="http://sourceforge.net/sflogo.php?group_id=429" width="88" height="31" border="0" alt="SourceForge Logo">
</a>
</center>
</body>
</html>
|
