File: lg_answer.html

package info (click to toggle)
lg-issue75 2-1
links: PTS
area: main
in suites: sarge
size: 1,100 kB
ctags: 212
sloc: perl: 228; makefile: 62; sh: 34
file content (709 lines) | stat: -rw-r--r-- 28,371 bytes
<!--startcut ======================================================= -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<META NAME="generator" CONTENT="lgazmail v1.4F.k">
<TITLE>Linux Gazette 75: The Answer Gang (TWDT)</TITLE><TITLE>The Answer Gang 75: </TITLE>
</HEAD><BODY BGCOLOR="#FFFFFF" TEXT="#000000"
	LINK="#3366FF" VLINK="#A000A0">
<!--endcut ========================================================= -->
<P>
<!--startcut ======================================================= -->
<CENTER>
<!-- *** BEGIN navbar *** -->
<IMG ALT="" SRC="../gx/navbar/left.jpg" WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="bottom"><A HREF="lg_tips.html"><IMG ALT="[ Prev ]" SRC="../gx/navbar/prev.jpg" WIDTH="16" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="index.html"><IMG ALT="[ Table of Contents ]" SRC="../gx/navbar/toc.jpg" WIDTH="220" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../index.html"><IMG ALT="[ Front Page ]" SRC="../gx/navbar/frontpage.jpg" WIDTH="137" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue75/lg_answer.html"><IMG ALT="[ Talkback ]" SRC="../gx/navbar/talkback.jpg" WIDTH="121" HEIGHT="45" BORDER="0" ALIGN="bottom"  ></A><A HREF="../faq/index.html"><IMG ALT="[ FAQ ]" SRC="./../gx/navbar/faq.jpg"WIDTH="62" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="lg_bytes.html"><IMG ALT="[ Next ]" SRC="../gx/navbar/next.jpg" WIDTH="15" HEIGHT="45" BORDER="0" ALIGN="bottom"  ></A><IMG ALT="" SRC="../gx/navbar/right.jpg" WIDTH="15" HEIGHT="45" ALIGN="bottom">
<!-- *** END navbar *** -->
</CENTER>
</p>
<!--endcut ========================================================= -->
<P> <hr> <P>
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<center>
<H1><A NAME="answer">
	<img src="../gx/dennis/qbubble.gif" alt="(?)" 
		border="0" align="middle">
	<font color="#B03060">The Answer Gang</font>
	<img src="../gx/dennis/bbubble.gif" alt="(!)" 
		border="0" align="middle">
</A></H1> 
<BR>
<H4>By Jim Dennis, Ben Okopnik, Dan Wilder, Breen, Chris, and...
	(<a href="tag/bios.html">meet the Gang</a>) ...
	the Editors of Linux Gazette... 
	and You!
<br>Send questions (or interesting answers) to
	<a href="tag/ask-the-gang.html">the Answer Gang</a> for
		possible publication 
</H4>
</center>
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<p><hr><p>
<H3>Contents:</H3>
<dl>
<dt><a href="#tag/greeting"
	><strong>&para;: Greetings From Heather Stern</strong></A></dl>

<DL>
<!-- index_text begins -->
<dt><A HREF="#tag/1"
	><img src="../gx/dennis/qbub.gif" height="28" width="50"
	  alt="(?)" border="0"
	><strong>How does one examine a core file</strong></a>
<!-- index_text ends -->
</DL>
<!--     .~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.     -->
<A NAME="tag/greeting"><HR WIDTH="75%" ALIGN="center"></A>
<H3 align="left"><img src="../gx/dennis/hbubble.gif" 
	height="50" width="60" alt="(&para;) " border="0"
	>Greetings from Heather Stern</H3>
<!-- begin hgreeting -->
<p><em>... you stand there waiting for Heather to look up from her keyboard...</em></p>
<p>Oh!  Hi everybody!  It's certainly been an active month here with The Answer
Gang.   We had almost 700 slices of Gazette related mail come past my inbox.
The longest thread (not pubbed this month, look forward to it next time) was
over 50 messages long.  Less than 20 people got no answer whatsoever (not 
counting the occasional spammer) and the top reason for not getting a post
answered, appeared to be simply a lack of interest in that message.   Crazy
attachments are down a LOT since our sysadmin improved the filters.  Ben did
a bit more cleanup on the <a href="../tag/members-faq.html"
	>TAG FAQ</a> and <a href="../tag/kb.html">Knowledgebase</a> and we
	have a new <a href="tag/ask-the-gang.html">posting guidelines page</a>
	which I hope you find easy to read.
<p>In the land of Linux I'm pleased to note that the 2.4 series kernel is
resembling stable since 2.4.17 is over a month old now.  A lot of work is being
done in 2.5.
<p>Flu struck my area and melted my mind back down to a mere single CPU when I'm
used to being an SMP system.  Bleh!  And before you ask ... yes,
I'm feeling better.  Lots of liquids, chicken soup, all that.
<p>It appears as though Ghostscript is my evil nemesis of the month.  
I haven't had time to finish compiling support for that new color
printer of mine.  In a moment of foolishness I upgraded my Dad-in-law's box
and the next few days were completely nuts since kword and gs refused to
agree on what fonts to print, or even to get the metrics right so margins
would work.  They're happy again since I forced ghostscript to uninstall
completely and then reinstall.   And we still wonder what the heck happened
to gnucash in Debian/Woody, though I admit, I haven't looked very hard.
<p>Cheerfully for my mortgage I've had a lot of consulting work this month.
Between 600 plus messages and all that, though, there wasn't time for me to
fit the usual ten pack (this blurb and nine of the juiciest TAG threads) in
under a tighter than usual deadline.  
Mike will be enjoying a Python conference much of this next month.  I hope it
counts for a well deserved vacation on his part.  
<p>I've not left you completely wanting, though.  Here's a few days in the 
life of The Answer Gang, troubleshooting one of those day to day things that
drives everybody nuts once in a while -- segfaults.
<p>Core files are a mess.  Good thing we have a dustbin around here.
<!-- end hgreeting -->
<!--     .~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.     -->
<A NAME="tag/1"><HR WIDTH="75%" ALIGN="center"></A>
<!-- begin 1 -->
<H3 align="left"><img src="../gx/dennis/qbubble.gif" 
	height="50" width="60" alt="(?) " border="0"
	>How does one examine a core file</H3>


<p><strong>From Faber Fedor 
</strong></p> 
<p align="right"><em>Answered By Jim Dennis, Dan Wilder, John Karns,
	<br>with side comments from Ben Okopnik and Heather Stern
</em></p>

<strong>
<p>
I've got a problem with a RH7.1 machine and no error messages to look at, so
I'm wondering how does one debug a problem like this?
</p>
<p>
Moved a machine from NY to NJ yesterdy.  When I left it last night,
everything was running, esp. <A HREF="http://www.apache.org/">Apache</A>.  This morning, normal maintanence
occurred at 4:02 AM, and when the system (syslog?) went to restart
httpd, the restart failed.  It's been failing ever since too!
</p>
<p>
The only http related message in <TT>/var/log/messages</TT> is
</p>
<pre>
Dec 22 12:27:13 www httpd: httpd startup failed
</pre>
<p>
Access and error logs for httpd are empty.
</p>
<p>
Running <TT>/usr/sbin/httpd</TT> (with and without command line parms) generates
the message
</p>
<pre>
Segmentation fault (core dumped)
</pre>
<p>
and the requisite core file:
</p>
<pre>
core: ELF 32-bit LSB core file of 'httpd' (signal 11), Intel 80386,
version 1, from 'httpd'
</pre>
<p>
File size and date of <TT>/usr/sbin/httpd</TT> matches my local copy.
</p>
<p>
Any ideas where to look next?
</p>
--
Regards,
Faber
</p>
</strong>

<!-- sig -->
<p><em>Jim Dennis pontificates about troubleshooting apache's startup... -- Heather</em></p>

<!-- end 2 -->
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
First, I would run <TT>/etc/init.d/httpd</TT> or <TT>/etc/init.d/apache</TT>,
or whatever it is on your system.  Run it with the "start"
option.
</blockQuote>
<blockQuote>
(Actually I'd <EM>read</EM> the <TT>/etc/init.d/</TT> start script for that
service, and probably I'd manually go through it to figure
out what I needed to do in order to run this particular installation
of Apache correctly).
</blockQuote>

<strong>
<p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
Did that.  That's what I meant by "it crashed at the command line with
and wothout parameters.
</p>
</strong>

<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
To dig further I might replace the httpd with a short
"strace wrapper" script:
</blockQuote>

<blockquote><pre>#!/bin/bash
exec strace -f -o /tmp/apache/strace.out /usr/sbin/httpd.real "$@"
</pre></blockquote>
<strong>
<p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
This <EM>definitely</EM> goes into my bag of tricks (once I 
<a href="#strace-trick">decode it</a>
<IMG SRC="../gx/dennis/smily.gif" ALT=":-)" 
		height="24" width="20" align="middle">)
</p>
</strong>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
(be sure to mkdir <TT>/tmp/apache</TT>, and make it writable to the
appropriate UID/GID --- whatever the webserver runs as).
</blockQuote>
<blockQuote>
I'd look through the strace.out file for clues.
Don't leave this running in this fashion for too long.  The
strace.out files will get <EM>huge</EM> very quickly; and your performance
should suffer a bit.
</blockQuote>
<blockQuote>
Considering that it used to work, you did a shutdown, moved the
system, brought it back up,  and then, presumably, CONFIGURED IT
FOR A NEW NETWORK, I'd look very carefully at network masks, routes
and related settings.
</blockQuote>
<strong>
<p><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
Very close!  The problem turned out to be that the name server the box
was using is no longer accessible (the box is there, but dig returns "no
name servers were found") and there were no backup name servers in
<TT>/etc/resolv.conf</TT> (mea culpa).
</p>
<p>
I wouldn't have expected apache to segfault under those conditions, but
it did.
</p></strong>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
Also, consider upgrading to RH7.2 if you can.
</blockquote>
<strong>
<p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Faber]
I just got my hands on it earlier this week so I'm still evaluating it.
</p>
</strong>
<blockquote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
  <A HREF="http://www.redhat.com/">Red Hat</A>'s distribution
has been very consistent in it's release history: avoid the .0,
skip the .1, and wait for the .2; that's been the rule since 4.2!
</blockQuote>

<strong>
<p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Faber]
Normally, that's what I do, but we needed to upgrade to PHP4 ASAP and it
was alot easier to upgrade the whole system to 7.1 (from 6.2).
</p>
<p>
thanks again!
<br>Regards,
Faber
</p>
</strong>

<blockQuote>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
You're welcome.
</blockQuote>
</blockQuote>

<hr width="40%" align="center">
<p><em>... while Dan took a different approach, considering the core file itself. -- Heather</em></p>

<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan]
0) Start by making sure there's no error in your httpd.conf by
running
</blockQuote>
<blockQuote><code>
apachectl configtest
</code></blockQuote>
<blockQuote>
No doubt there's nothing there.  But if there is, you are not
apt to find it by examining core files, etc.
</blockQuote>
<blockQuote>
If you're an expert C developer
</blockQuote>
<strong>
<p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Faber]
At one point in my life, I might have said that, but then only to
impress women like Heather. 
<IMG SRC="../gx/dennis/smily.gif" ALT=";-)" 
		height="24" width="20" align="middle">
</p></strong>
<blockquote>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan]
I don't expect Heather's that easily impressed.  Especially by guys
like me that mistype "developer".
</blockQuote>
<blockquote><em>That's ok, I fixed it.  That's what editors are for,
	at least sometimes.  I'm more impressed by how people solve
	problems than by whether they're an expert in everything around
	them.  It's nice if they can solve <strong>my</strong>
	problems, though. -- Heather
</em>
</blockQuote>
</blockQuote>
<blockquote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan]
 and have the source tree to your
apache handy, examining the core file might yield you something.
</blockQuote>
<strong>
<p><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Faber]
IOW, no, I don't want to do that. 
<IMG SRC="../gx/dennis/smily.gif" ALT=":-)" 
		height="24" width="20" align="middle">
</p>
</strong>
<blockquote>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan]
Naah, me neither.  Last resort.
</blockQuote>
</blockquote>
<blockquote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
Mostly it's pretty indirect.  Segfaults are typically caused by
out-of-bounds pointers or array references, references to allocated
memory since freed, confusion about number or type of parameters
passed to a function, and the like.
  The error happens earlier,
when the bad pointer is parked someplace, memory is erroneously
freed, etc.  The fault happens later, when something is dereferenced.
</blockQuote>
<blockQuote>
I've spent many a happy and well-paid hour trying, sometimes
without success, to track backwards from fault to error.
And when you find the error, you may still a long and
winding road back to the defect which caused the error.
</blockQuote>
<blockQuote><pre>
Defect    ---------&gt;  Error  -------------&gt; Fault

(Improper          (Something bad       (Result becomes
code construct)     happens)             observable as
                                         unexpected result)
</pre></blockQuote>
<blockQuote>
Unless you're an expert C developer, and patient and lucky
as well, it's more likely you'll find the problem by a process
of elimination.
</blockQuote>
<blockQuote>
1) What's changed recently?  New application?  Change in httpd.conf?
New module installed?  Try backing out any recent changes,
one by one.  Restart apache after each thing you back out.
</blockQuote>
<blockQuote>
2) Is it possible there's filesystem corruption?  Corrupted binaries
often fail to run well.  Take the machine down and run
</blockQuote>
<blockQuote><CODE>
fsck -f
</CODE></blockQuote>
<blockQuote>
on all filesystems.  If you find anything amiss, determine
what files were affected.
</blockQuote>
<blockQuote>
3) Reinstall apache just in case, anyway.
</blockQuote>
<blockQuote>
4) Could the machine have other hardware problems?  If you have
the kernel development packages installed, build the kernel
eight or ten times.  If you get "died with signal 11" or other
abnormal termination, proceed with hardware troubleshooting
procedures.
</blockQuote>
<blockQuote>
5) Figure out what area of apache is affected.  Save your httpd.conf
and start with a default one.  Will apache start?  If so,
re-introduce features from the running copy of httpd.conf
a few at a time until apache begins dying at startup.
</blockQuote>
<blockQuote>
Let us know how you do.  Depending on where you find trouble,
the gang can offer further advice.
--
Dan Wilder
</blockQuote>

<hr width="40%" align="center">
<p><em>Jim has quite a bit to say about 
<a name="strace-trick">using strace</a> -- Heather</em></p>

<blockquote><pre>#!/bin/bash
exec strace -f -o /tmp/apache/strace.out /usr/sbin/httpd.real "$@"
</pre></blockquote>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
In runs a shell (bash) which then exec()s (becomes) a copy of
the strace command.  That strace command is told to "follow forks"
(so we can trace the system call of child processes) and writes
its output to a file in our <TT>/tmp/apache</TT> directory.  strace then
runs (fork()s then exec()s) a copy of the "real" httpd with a
set of arguments that matches those that were passed to to
our script.
</blockQuote>
<blockQuote>
The distinction between <TT> exec()</TT>'ing a command and invoking it in
the normal way is pretty important.  Normal command invocation
from a UNIX shell involves a <TT> fork()</TT> (creating a clone process
which is a subshell) and then an exec*() by that shell to transform
that subprocess into one which is running the target command.
</blockQuote>
<blockQuote>
Meanwhile the parent shell process normally does a wait*()
on the child.  In other words, it sits there, blocked
until the child exits, or until a signal is received.
</blockQuote>
<blockQuote>
When we use the shell exec command, it prevents the <TT> fork()</TT>
(there's no creation of a subprocess).  The "text" (executable
binary code) of the process that was running a copy of your
shell (<TT>/bin/bash</TT> in our case) is overwritten by the "text"
of the new program; all of the heap and stack segments (memory
blocks) of the old process are freed and/or clear) and the only
traces of the old memory image that remain available are the
contents of the process' environment.  In other words, the
exec command is a wrapper around the one of the exec*() system
calls (there are several different versions of the exec*() system
call which differ in the format of their arguments, and the
preservation/inheritance versus creation of environments).
</blockQuote>
<blockQuote>
Actually I think that Linux kernel implements <TT> execve()</TT> as a
wrapper around its <TT> clone()</TT> system call, and that libc/glibc
provides the handling for all of the variations on that.  The
three "variables" on these exec variations are:
</blockQuote>
<blockQuote>
<dl><dt>
format of the command argument list: <dd>(which is either
done through C varargs --- like <TT> printf()</TT> and friends,
or is a pointer to an array of NUL terminated strings),
(execv* vs. execl*)
<br>&nbsp;
<dt>
environment handling: <dd>whether the process keeps its
current environment or overwrites it.  The <TT> execle()</TT>
and <TT> execve()</TT> versions have an extra parameter pointing
at an NUL terminated of NUL terminated strings.
<br>&nbsp;
<dt>
path searching: <dd>The first argument of the <TT> execvp()</TT> and
 <TT>execlp()</TT> functions can be a simple command basename
--- while all other variations require a qualified
path.  The "p" versions will search the PATH as a
shell would.
</dl>
</blockQuote>
<blockQuote>
It appears that you can either search the PATH or create a
new environment, but not both.  Of course you can use a simple
 <TT>execl()</TT> or <TT> execv()</TT> to do neither.  Of course you can read the
man exec(3) manual pages in the library functions section of
your online docs to read even more details about this.
</blockQuote>
<blockQuote>
When I'm teaching shell scripting I spend a considerable
amount of time clarifying this worm's eye view of how UNIX
and the shell handles fork()s and exec*()s.  I draw diagrams
representing the memory space and environment of a process,
and another of a child process (connected by dotted lines
labeled  <TT>"fork()</TT>").  The I crosshatch most of the memory space
--- leaving the environment section, and label that exec*().
</blockQuote>
<blockQuote>
When I do this, people understand how the environment really
works.  The "export" shell command moves a shell variable
and its value from the local heap "out" to the environment region
of memory.  Once they really understand that, then they won't
get too confused when a child process sets a shell variable,
exports, and then their original process can't see the new
value.  ("export" is more of a memory management operator than
an inter-process communications mechanism; at best it is a
one-way IPC, <EM>copying</EM> from parent to children children).
</blockQuote>
<blockQuote>
After than I generally have to explain about some implicit
forms of sub-process creation (forking) that most people
miss.  In particular I remind them that pipes are an
*inter-process* communications channel.  So, any time you
see or use a | operator in the shell, you are implicitly
creating sub process.  That's why a command like:
</blockQuote>
<blockQuote><CODE>
unset bar; echo foo | read bar; echo $bar
</CODE></blockQuote>
<blockQuote><blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Ben]
Oh, <em>that's</em> cute. I go through pretty much the same spiel - some of it
admittedly cribbed from your description of this, because I liked it the
first time I heard it - but the way I've been demonstrating it is with a
</blockQuote>
<blockQuote><code>
while read bar; do echo $bar; done &lt; file
</code></blockQuote>
<blockQuote>
loop. This nails down the other end. <EM>Very</EM> cool.
</blockQuote>
<blockQuote>
(Scribbling notes in newly acquired Palm Pilot)
</blockQuote></blockQuote>

<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
... will return an empty value in most shells.  The read
command is executed in a subprocess which promptly exits,
freeing the memory that held <EM>its</EM> copy of the bar
variable/value pair.  (I say <EM>most</EM> shells because ksh '93
and zsh, create their subprocesses on the left hand side
of their pipe operators.  That's one of those subtle differences
among shells.  Personally I think bash and others do it wrong,
the ksh/zsh semantics are superior and I hope bash 2.x or 3.x
will adopt them, or offer a shopt, shell option, to select the
desired semantics).
</blockQuote>
<blockQuote>
The "<tt>$@</tt>" ensures that the arguments that were passed to us wil be
preserved in count and contents.  If we used "<tt>$*</tt>" we'd be passing
a single argument to our command.  That single argument would
contain the text of all of the orginal arguments, concatenated
as one string, separated by spaces (or by the first character from
IFS if you believe the docs).  If we used <tt>$*</tt> (no soft quotes)
we'd be having the current shell resplit the number of arguments
--- they'd have the same contents, but any arguments that had
previously had embedded spaces (or other IFS characters) would
be separated accordingly.
</blockQuote>
<blockQuote>
The "<tt>$@</tt>" handling is the most subtle part of this script.
An unquoted <tt>$@</tt> would be be the same as an unquoted <tt>$*</tt> (as far
as I can tell).  It is just the "<tt>$@</tt>" that gets the special
handling. (<tt>$*</tt> and "<tt>$*</tt>" aren't special cases, they 
are expanded and split in the normal way; "<tt>$@</tt>" is expanded and 
sort of "internally requoted" to preserve the <tt>$#</tt> --- argument count).
</blockQuote>
<blockQuote>
If you were going to need to do this frequently we might
write a "strace.wrapper.sh" shell script which would work
a bit like this:
</blockQuote>

<blockquote><pre> #!/bin/bash
 OLDMASK=$(umask)
 umask 077
 TMPDIR=/tmp/$(basename $1)$$
 mkdir "$TMPDIR" || exit 1
  ## make a temporary directory or die
 umask $OLDMASK
 TARGETCMD="$1"
 shift
 exec strace -f -o "$TMPDIR/strace.out" "$TARGETCMD" "$@"
</pre></blockquote>
<blockQuote>
In this example we call strace.wrapper.sh with an extra
argument, the name of he command to be "wrapped."  We then
fuss a little with umask (to insure that our process' output
will have some privacy from prying eyes, and doing an atomic
"make a private dir or die trying" (This is the safest temp
file handling that can be managed from sh, as far as I know).
</blockQuote>
<blockQuote>
Then we restore our umask, (so we don't create a Heisenbug by
challenging one of our target command's hidden assumptions about
the permissions of files <EM>it</EM> creates).  We than grab our target
command, shift it off our argument list (which does NOT disturb
the quoting of the remaining arguments) and call our strace
command as before --- with variables interpolated as necessary.
</blockQuote>
<blockQuote>
Mind you I don't use this script.  I don't bother since I can
do it about as easily by hand.  Also this script wouldn't be
the best choice for CGI, inetd launched, or similar cases.
In those cases we're better renaming the original binary.
</blockQuote>

<hr width="40%" align="center">
<p><em>Of course we were all happy when Faber found what it was!  We encouraged
	him to send in his bug report -- Heather</em></p>

<strong>
<p>
I wouldn't have expected apache to segfault under those conditions, but
it did.
</p></strong>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
Report it as a bug (after upgrading to the latest stable release).
Try to isolate the .conf directive(s) that are involved, if possible.
</blockQuote>

<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan] ...  The error happens earlier,
when the bad pointer is parked someplace, memory is erroneously
freed, etc.  The fault happens later, when something is dereferenced.
</blockQuote>
<strong><p><IMG SRC="../gx/dennis/qbub.gif" ALT="(?)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	>
Well, as I told Jim, the fact that it couldn't find a name server
caused it to segfault.  Weird; you would have thought it would have
exited wih a message at least.
</p></strong>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [John K]
It sounds like there's a bug or some abnormality with apache's handling of
a situation which is doesn't expect in normal operation.  IOW, a problem
with error handling.  If the apache version is not the latest stable
version, you might want to consider upgrading.  If it is the latest, then
you may want to consider reporting it to the apache developers.
</blockQuote>

<hr width="40%" align="center">
<p><em>...and of course we congratulated him on his success, with some extra
	thoughts on general troubleshooting. -- Heather</em></p>

<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [Dan]
Congradulations on solving the problem.
</blockQuote>
<blockQuote>
That's what I call the "natural history approach".  Examine carefully the
behavior and habitat of the creature in question, and think carefully
about what you've observed.
</blockQuote>
<blockQuote>
I've probably fixed a lot more bugs in my life by the natural history
method, than I have by the method of examining core files, or for that
matter running under a debugger or emulator.
</blockQuote>
<blockQuote>
Strace, mentioned separately in this thread, is a little harder to classify.
A program that attaches itself to a running process and dumps out information
about system calls, it affords a level of information about a program
that may sometimes come close to what you'd see using a debugger.
</blockQuote>
<blockQuote>
Mostly it doesn't, but sometimes it provides that key observation not
available by other means which allows us to finally come to grips with
a bug.  I'd group it with natural history tools, perhaps as an analog
to a radio collar.  You know where the animal's been, but maybe not why,
or what it did there.
--
Dan Wilder
</blockQuote>
<blockQuote><IMG SRC="../gx/dennis/bbub.gif" ALT="(!)"
	HEIGHT="28" WIDTH="50" BORDER="0"
	> [JimD]
I like to use the classic "OSI reference model" as a rough
troubleshooting sequence.  Keep going down the stack (from
application, down through network and to the physical layers
until you isolate the problem, then proceed back upwards
correcting each problem until the application works).
</blockQuote>

<!--     .~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.~~.     -->
<P> <hr> </p>
<!-- *** BEGIN copyright *** -->
<H5 align="center">This page edited and maintained by the Editors
        of <I>Linux Gazette</I>
<a href="http://www.linuxgazette.com/copying.html"
        >Copyright &copy;</a> 2002
<BR>Published in issue 75 of <I>Linux Gazette</I> February 2002</H5>
<H6 ALIGN="center">HTML script maintained by
        <A HREF="mailto:star@starshine.org">Heather Stern</a> of
        Starshine Technical Services,
        <A HREF="http://www.starshine.org/">http://www.starshine.org/</A>
</H6>
<!-- *** END copyright *** -->
<!--startcut ======================================================= -->
</BODY></HTML>
<!--endcut ========================================================= -->