My flight leaves tonight, so I spent the day wandering around central Brisbane. Ben LaHaise happened to take the same CityCat ferry (which is a system of catamarans used to ferry people up and down the Brisbane River) as me down to the city. He tried to blow up the boat with his umbrella, but, fortunately, an alert crew member stopped him.

Back to the Uni at the end of the afternoon, collect my bags, call a taxi to the airport, and wait for the flight to Seoul. Then it's another 14 hours to JFK and another couple back to CT and then I will be hopelessly confused about what time it is for a couple of days.

I'm off to Australia tomorrow for LCA 2002. This caps a fairly productive week of UML bug hunting:

• mistral and blinky started seeing a panic in fork. mistral figured out how to reproduce it and tracked it down to the point where it had something to do with kernel threads reference counting their mm's. This was enough information for me to fix the bug, by giving kernel threads NULL mm's.
• While I was at ISTS giving a talk, I spent some free time looking at the pthreads problem that people have been seeing for a while. It turns out to be a problem with UML signal delivery. I had assumed that the registers going into a signal handler didn't matter (except for the IP and SP, of course) and that only the stack frame mattered. So, the process registers at the start of a signal delivery are initialized with a set that was captured from a UML thread at boot time. This works fine usually, except that recent pthreads libraries store some thread-private data in %gs, and the %gs value has to be preserved in signal handlers. So, the UML signal delivery mechanism needs to be reworked again.
• The UML IO hangs that were reported this week were tracked down and found to be a bug in the host's handling of SIGIO. It turns out to be possible, on an SMP host, for SIGIO to be queued to a process after that process has returned from the fcntl that registered a different process as the SIGIO recipient. This breaks UML badly because SIGIOs end up queued, but not delivered, to a process which is out of context and sleeping.

The security work is largely done. The exception is the lcall prevention fix that's needed on the host. So, I released 2.4.17-9 today. It also contains a number of fixes and patches from other people, the largest being the latest set of James McMechan's ubd changes.

I made another patch against the 2.5.2-pre tree again today. This one is against 2.5.2-pre11. It has been sent off to Linus so he can drop it in his bit bucket.

In another development, it turns out that the O(1) scheduler breaks UML by holding IRQs disabled across context switches. This results in SIGIO (i.e. from disk IO completions) to be trapped in a process that has gone out of context, and can't be woken up until something else notices that the IO has completed, and of course it won't because the SIGIO has been delivered to the wrong process.

After spending five days tracking down a swap corruption bug, I discovered that Rodrigo de Castro had explained to me exactly what it was about a month ago. Unfortunately, at the time, I didn't know enough about the swap code to decide whether he was making sense. Of course he was, and I discovered that at the end of the great bug hunt.

So, with that fix, Lennert's latest SMP changes, and a bunch of smaller stuff, I'm releasing 2.4.17-5 today.

Linus saw fit to silently drop UML into the bit bucket again, so I'll make another patch soon and send it in.

I released the 2.4.17-3 and 2.4.17-4 patches this week. The biggest change has been the merging of Lennert's SMP fixes.

I made another attempt to get UML into the Linus tree. This patch is against 2.5.2-pre9 and is the 2.4.17-4 patch. We'll see how well this attempt fares.

I announced the full 2.4.17 release and the 2.4.17-2 patch today. The patch is largely changes to allow UML to calculate current from the stack. This is to make life easier for Lennert, who's trying to get SMP working. It also contains a bunch of fixes for bugs that crop up when host devices get closed from under UML consoles or serial lines.

Iain Young is making a decent stab at a UML/sparc64 port. He's got the boilerplate filled in (albeit with some skeptical comments about their correctness...) and he's trying to get the whole thing to compile.

Linus released 2.5.2-pre4 today with no sign of UML in the changelog. I grabbed the patch to make sure it wasn't there. It wasn't. Grrr. I'll give him another patch and then I'll spam him with it again.

OK, so the diary has taken a bit of a holiday break. I released the 2.4.17 UML patch yesterday. The full release will be forthcoming. I also ported that patch into 2.5.1, created the 2.5.1 patch, and sent it to Linus. Hopefully, he will put it in without my having to resend it too many times. I sent a little note off to LKML announcing this, which got some favorable reaction, both on and off the list. Alan, being his usual taciturn self sent a reply, which read, in its entirety, "Cool".

This release had a lot of accumulated stuff in it. The biggest item are the port channel, which let you attach any number of UML consoles and serial lines to a host port, at which point you can access them by telnetting to that port. I also redid the context switching mechanism after thinking of a much simpler way of doing it. This should also be much faster since it doesn't involve signals flying around, so context switches are now invisible to the tracing thread.

Back from Linux-Kongress. Back on US/Eastern. I think I never left it, which made life (the staying awake part of it) difficult in Holland.

As for highlights of the conference, we have:

• I and the UML project seem to have some name recognition. Everyone I talked to seemed to have heard of both me and UML, which is very cool.
• The talk went reasonably well. I gave the same talk as I did at ALS. I had forgotten that it was somewhat tailored for ALS (i.e. I tried to avoid talking about stuff that I had talked about at the previous ALS), and it would have been somwhat different if I had prepared a new talk for Linux-Kongress.
• I met a pile of cool people, like
  • Lennert Buytenhek (who, in recognition of his contributions to both projects, was embarassed by both me and Rusty by being asked to stand up for some audience appreciation, which must be a new Linux-Kongress record)
  • Roman Zippel (who told me about a couple of ubd driver bugs, one of which I knew of (a subtle rounding error), and one of which I didn't (Greg Lonnon and I went to some trouble to put the COW header in network byte order, but forgot to do the same for the block bitmap (grrrr))
  • Bruce Walker (who's in charge of the Compaq SSI project, and who asked me clustering questions during my talk, at which point, I (correctly) guessed who he worked for and why he was asking)
  • Fabio Olive Leite (a Conectivite who gave me a nice Conectiva filesystem image a while back, and whose name I unaccented to get it through my XSL processor)
  • Philipp Reisner (who threatened an Alpha UML port a while back, but gave up, so he's less cool than the others :-)
• The organizers took the speakers back to Amsterdam after the conference to spend the day bumming around the city. We broke up into small groups and went our separate ways. Our group spent much of our time in two smallish cafe-type places just talking about random stuff.
• The train trip from Schiphol airport (Amsterdam) to Enschede (near the German border) was interesting for a number of people. It was exactly as described for me (2+ hours, direct train, no problem), but some track maintenance in Amsterdam's central station caused subsequent trains to be cancelled, so other people had nightmares involving 5 trains and a bus totalling 5 hours.

I released 2.4.14-6 today in the interest of clearing the decks for the 2.4.15 release. Before leaving for Thanksgiving, I had redone the mconsole protocol to be packet-oriented. This allows a lot more flexibility in what can be done with the protocol. As a result, you'll need the new mconsole client for this release.

While down in CT, I redid the host channel support. It is all much cleaner now, and makes it a lot simpler to knock a bunch of related items off the todo list.

I decided to knock off the 2.4.15 patch today as well. It went cleanly, aside from a ptrace cleanup and a new way of generating /proc/cpuinfo which I had to support. I also put in the file corruption fix after forgetting to and discovering that a boot/halt caused fsck to complain a little.

While I'm waiting for the meteors to arrive, I'm chasing and stomping UML bugs. I cleaned up and released the proxy arp fixes that I did on planes and in airports on my way to Oakland. Before, uml_net would blindly add an arp entry to eth0 and nothing else. This is wrong if there is no eth0, and it's also wrong if eth0 doesn't connect to the local net or if there are other interfaces also attached to the local net. uml_net now looks at the routing table and puts an arp entry on every interface that talks to the local net.

I also noticed that slip support wasn't up to date, so I modernized it and cleaned up the code while I was at it. You can now change the IP address of a slip-based interface and the host configuration will be updated just like the other transports.

I added some RT signal support. SA_SIGINFO is now supported, which will hopefully fix some of the strange process behaviors that have cropped up lately. If this fix doesn't do it, I chased down another bug which was causing rt_sigsuspend and sigsuspend to return incorrect values. This was causing the libc sigsuspend to hang, and its process with it. This fixes the pthread_create hang that Greg Lonnon noticed, plus the gdb hang, I think. I haven't checked that yet.

Those fixes are in 2.4.14-3um which I just released. You'll need the latest utilities in order to use the network, since I bumped the uml_net version again.

In preparation for fixing the problem of the console driver losing output, I ported the SIGIO handler to use poll instead of select. This was mostly what 2.4.13-4 was. I later discovered a bug in it, which is fixed in -5.

I then decided to fix the problem of UML not being able to be interrupted and backgrounded. The problem was that all UML processes are in the same process group, with all of them stopped except for the one that's actually running. The problem is that when UML is backgrounded, the shell sends a SIGCONT to the process group, which wakes up every UML process, which is very bad.

I did some failed experiments with setpgrp/setsid and friends, and discovered that a separate process group wouldn't work because then those threads can't write to the terminal because they're in the wrong process group.

So, I decided that out-of-context processes should be asleep rather than stopped. This required redoing the task switching code. They were stopped because the tracing thread intercepted a signal from them when they went out of context and never continued them. Having them sleep would require that the tracing thread stop doing that and that the threads involved in a context switch arrange the transfer themselves.

So, what is now done is that non-running processes are asleep in sigsuspend, and they are woken up by the going-out-of-context process sending a SIGTERM. Races are avoided by having the SIGTERM sent inside a section of code that has blocked SIGTERM. SIGTERM is re-enabled atomically with the sleep with sigsuspend.

So, that plus the poll fix is the contents of -5.

That last patch went into -ac6, so the ac UML builds and works again. The next job is to get the ac tree up to date.

I released a new utilities tarball today. uml_net should now do proxy arp correctly. uml_mconsole is now able to take a command on its command line and execute it, rather than being strictly a command line tool.

Today is 2.4.14-3 day. I decided to remove the code in fix_range which unmaps pages whose ptes say they're not present. That basically caused it to try to uselessly unmap all of its unused address space. So, I did that and it uncovered a bug. It turns out that swapped-out pages weren't marked as needing to be remapped. Everything worked a lot better with that fixed, and context switching should be a bit faster now.

I released 2.4.14-2 today. This contains the fix for the process segfaults and the gdb problems people have been having. It also turns on morlock's context switch optimization which I disabled until I figured out the segfaults.

I finished releasing 2.4.13 today.

After some prodding from Greg Lonnon, and after he did some investigation, I figured out what the problem with gdb inside UML is. The signal handlers don't save their registers in the thread struct. This means that when a SIGTRAP from a breakpoint comes in and it gets forwarded to gdb, when it gets the registers to find out what the ip is, it gets an old, bogus value. So, it doesn't recognize that as a breakpoint and complains about a spurious SIGTRAP instead.

I spent the last few days chasing a process segfault problem. I finally tracked it down today. It turns out that my rewrite of the process signal delivery code was broken in the case of a signal being delivered from an interrupt handler rather than a system call. It grabs the process registers from the thread structure, saves them away on the stack, and then restores them to the process when the handler finishes.

However, interrupts don't save their registers in the thread structure, so those registers represent the last system call, which has already finished. And restoring those causes great confusion in the process.

I released 2.4.12-3um and 2.4.12-4um over the last week. -3 fixed a couple of problems with -2, and -4 adds some miscellaneous fixes to that. The major ones are that physical memory protection is optional (controlled by the 'jail' switch) and that the network driver backends now collect uml_net commands and output and nicely printk them instead of having the output just dumped to the terminal. To support this, uml_net now hangs on to the commands it runs and the output they produce and send them back to UML. This required that the uml_net interface be incremented, so it's now at 3. The new drivers require the new uml_net, so if you grab the UML patch, also get the latest utilities tarball too.

I released 2.4.12-2um today. It's almost entirely changes sent in by other people, dominated by Adam Heath's cleanups. There were also some ppc fixes from Chris Emerson, and small fixes from other people.

I also released a new utilities tarball. The one change was to uml_net, which does proxy arp in a different, and apparently more robust way than it used to.

Linus released 2.4.11 and 2.4.12 two days apart. I had the 2.4.11 patch uploaded, and had started releasing packages when 2.4.12 came out. So, 2.4.12 is out there and I'm doing the packages again.

Paul Larson found a test case for the signal problems that was reproducable for me. So, with that in hand, I tracked down the bug, and released 2.4.10-6um.

The bug turned out to be a result of moving where state is saved before a signal is delivered to a process. The process registers and some other things need to be saved on the process stack so they can be restored later. The way it used to work is that

• handle_signal would figure out what the interrupted system call eventually returns
• that value is passed up the stack and stored in the process registers stored in its task structure
• the process would be sent a signal so it starts running on its process stack
• the UML signal handler copies the register state from the task structure to its own stack
• it calls the process signal handler
• and restores the registers back to the task struct

What I implemented did this

• handle_signal figures out what the interrupted system call eventually returns and constructs the process stack frame, copying the registers from the task struct onto the stack
• that value is passed up the stack and stored in the process registers stored in its task structure

The bug is that the second step happened too late. The registers saved on the stack hold a bogus return value, and it's that value which the system call eventually returns.

Today I redid the signal delivery code. Now all the saving and restoring of state happens in kernelspace rather than on the process stack like before. This allows the task structure to be protected from processes. Since that was the only hole in the protection of physical memory, that is now fully protected against being changed from userspace.

I made a .deb and an RPM in preparation for releasing 2.4.10, and Jacques Nilo reported that yesterday's fix wasn't enough. I had forgot about an instance of MAP_SHARED | MAP_ANONYMOUS. So, I fixed that, and that is 2.4.10-3um. And that is the basis of the official 2.4.10 UML release.

I thought of an easy fix for the stack capturing problem that prevented UML from booting on 2.2 hosts. Basically, a new process is created which stops itself, and when that happens, the parent grabs a copy of the stack and uses it to create a context for future threads to run in. On 2.2, the parent used ptrace to extract the contents of the stack from the child word by word. I looked at that code and decided it would be much easier to map the stack MAP_SHARED so it would be shared between parent and child and the parent could just memcpy it to a safe place rather than ptracing it out.

What I forgot was that, while 2.4 supports MAP_SHARED | MAP_ANONYMOUS, 2.2 doesn't. So, on 2.2 hosts, UML wouldn't even begin to boot.

The easy solution was to go back to MAP_PRIVATE | MAP_ANONYMOUS, but clone the new process with CLONE_VM, making it a thread, which allows the parent to copy the stack directly, since they're both in the same address space.

This fix makes 2.4.10 usable, so I've released another patch and updated CVS.

Linux released 2.4.10 today, so I updated UML as well. I decided not to base this on the latest UML patch, since that it not entirely healthy at the moment. My sigaltstack fixes broke UML totally on 2.2 hosts. So, 2.4.10-1um is 2.3.9-8um updated to 2.4.10.

CVS is not updated, but will be once I have the sigaltstack thing fixed and that pool updated to 2.4.10.

The last set of -ac patches went into -ac14. That brings Alan's tree reasonably up-to-date.

I released 2.4.9-8um yesterday and 2.4.9-9um today. -8 was some bug fixes and cleanup. -9 was fixing sigaltstack and doing a lot of cleanup and rearrangement of the signal delivery code. This sets me up to redo the entire signal delivery mechanism so I can finish protecting all of the kernel's physical memory from userspace.

I tracked down the process segfault problem. It was caused by a newly forked child inheriting some pages that were swapped out, but hadn't been unmapped. The code that it ran on its first quantum didn't update its address space correctly, so those pages remained mapped.

Having chased that problem down, I'm releasing 2.4.9-4um with that fix plus Chris Emerson's latest ppc changes.

Thanks to what looks like an all-night debugging session on the part of Yon Uriarte, the TUN/TAP backend now works. You'll need the latest uml_net for this. It wasn't setting IFF_NO_PI, which was causing extra cruft to be stuck on the front of the packet, which probably required the broken nastiness I had to add to the driver. Adding that and backing out all the skbuff fiddling made everything work a lot better.

So, I released 2.4.9-3um with the fixed driver in it, plus new entries in config.release, defconfig, and Configure.help.

I implemented a TUN/TAP backend for the network driver. It involved more work than I expected. A lot of it was due to restructuring other code in order to keep the code relatively clean.

I haven't done any stessing or timing of it, but I did happen to notice that pings over TUN/TAP are about 10x faster than pings over ethertap. The absence of the helper handling each packet on the way to the kernel is no doubt a big piece of that. At some point, I'll do some bandwidth measurements against ethertap to see how much better it is. Hopefully a lot.

I've been having fun playing with crashme. It's a great little tool. It generates buffers full of random data and then executes them. It runs differently on UML than on the host, which it shouldn't. The problems I've tracked down so far are signal handling bugs. UML wasn't handling write faults correctly when the accessed memory was readonly, and it wasn't properly segfaulting processes to which signals couldn't be delivered (because their stack pointers were garbage). This last was the bug I was chasing a couple days ago. There are still problems. The first process (crashme +2000 666 100) runs just as it does on the host, but the next one (crashme +2000 667 100) doesn't. On the host, the segfault handler somehow gets bus errors in libc, which I don't understand, and that doesn't happen under UML.

On IRC yesterday, Lennert Buytenhek clued me in on how to reliably segfault processes and crash UML. He was running 8 "du /". That didn't work for me, but 16 of them does. The segfaults are on pages that are mapped in but shouldn't be (their ptes say that they should be mapped out, and somehow that didn't happen). So, those pages were presumably allocated for something else, and contain garbage from the perspective of the process that should have unmapped them, and so it segfaults.

The panic looks like memory corruption. I turned on slab debugging, and it looks like that makes the panic go away.

Well, Linus released 2.4.9 today, so it's time for me to go into my UML release routine. When I did the obligatory kernel build on 2.4.9, one of the crashme fixes turned out to be bogus. It was doing the segfault-during-signal-delivery check too early, so it caught fixable segfaults that happened because the stack needed extending or was readonly.

2.4.8-2um is out as of yesterday. I made the freshmeat announcement of 2.4.8 this morning.

I chased the crashme bug a little. Somehow, a signal is marked as being pending, but it's never actually delivered and reset. So, no further signals can be delivered to that process from then on. This makes it unkillable and unstoppable.

I also took the first step towards making UML secure against nasty users. UML physical memory, except for the task structure and kernel stack, are protected from userspace access. I still need to protect the task structure and kernel virtual memory. The task structure is a bit tricky because of the signal delivery code. It runs on the process stack and is considered to be userspace code. However, it needs to be able to modify the task structure to restore state that it saved before the signal delivery. So, if the task structure isn't writable, this isn't possible. Further thought on the subject is necessary.

Well, Linus released 2.4.8 just as I was heading up north for a weekend of camping and climbing mountains. He does this on purpose. He released 2.4.3 when I had just arrived in San Jose for the Kernel Summit.

Anyway, this was a relatively simple patch. It just dropped in and worked, except that hostfs was already broken. My calculation of the stat64 inode field was wrong. It looked at the kernel version to decide what was in the userspace headers. I discovered the error of my ways when I booted up a Debian UML to produce the 2.4.8 .deb. This is a 2.2 filesystem (with .st_ino in stat64) with a 2.4 kernel (which implied .__st_ino in stat64). hostfs did not build. I changed the Makefile to just grep the appropriate header instead.

So, this fix will be the substance of 2.4.8-2um.

The remaining differences between my pool and the ac tree are a couple of patches that didn't go in for some reason, cleanups of printks and some includes.

Daemonizing UML does work. I just checked it, and the only case where it does something strange is if you background it without nohupping it and log out. The tracing thread dies from the SIGHUP, but all the other threads survive.

I released 2.4.7-5um today. It contains a few recent patches from other people. I figured out how to turn -fno-common back on. I tried all kinds of linker tricks to throw errno.o out of the binary. Then I discovered that the linking that had already taken place had destroyed any notion of what objects anything originally came from. So, instead, I added -Derrno=kernel_errno to all the kernelspace gcc lines, which translates all the kernel uses of errno to kernel_errno, and leaves the libc errno alone. That's actually a better solution than throwing out one of the errnos because that would leave open the possibility that the kernel and userspace uses of libc could step on each other. Now that they're using different symbols, that's not a problem.

Yesterday's patches are off to Alan.

In other news, daemonizing UML seems to be broken again. Grrr. That seems to break now and then for no apparent reason.

ac9 is out with my patches in it. So, time to make the final diff between Alan's stuff and mine to get him totally caught up with me.

The deb build problem turned out to be me accidentally redefining VERSION in the upper layers of the build process. That value overrode a VERSION in the kernel build, which resulted in a totally bogus KERNELRELEASE, which confused a macro which tested it badly enough that it broke the build. Simple to fix once I figured it out.

I discovered another hostfs bug on my way back from OLS. ls didn't work and I found two bugs as a result. The easy one was that hostfs_readdir was filling in the directory inode rather than the file inode for every directory entry it passed back to vfs. This was fixed by having read_dir pass the inode back up so it could be use to fill in the entry properly.

The more interesting one is that there was a source-incompatible change made in the stat64 struct between 2.2 and 2.4. The st_ino field changed its name to __st_ino and a new st_ino field was added at the end. The inode appears in the same place (the st_ino/__st_ino field) making it binary compatible. So, after changing to use the 2.4 field name (and breaking hostfs on 2.2), I changed the hostfs build to figure out what name to use and passing that in on the compile line to hostfs_user.c.

In other news, we (me, Rodrigo de Castro, and Livio Baldini Soares) have decided that -pg support in gcc is broken in multiple ways. rcastro and livio complained a couple weeks ago about UML's gprof support not working. I finally had a look at it, and found that it was broken, but not in the way they described.

UML crashed in a very inconvient place, and when I finally got in there enough to figure out what was happening, it turned out that mcount was segfaulting when it dereferenced ebp because ebp was NULL. The reason for that turned out to be that in some procedures, mcount is absolutely the first thing they do. Everything else calls mcount after the new stack frame has been set up and ebp has a valid value in it (the old esp). When the procedure is the main procedure for a thread, then ebp turns out to be NULL.

The difference between the two sets of procedures seems to be that the good ones have local variables and the bad ones don't. So, to work around this bug, I added a useless, but non-optimizable, local to the affected trampolines.

Having done that, rcastro and livio were still complaining about UML crashing. So, I looked at it with rcastro using gdbbot (and livio did so later and discovered the same thing). -pg was trashing edx for some reason. A constant (which varies from procedure to procedure) is dumped into it. This suggests that it's used for the profiling bookkeeping somehow, but looking at the assembly, we don't see how. mcount carefully pushes it and restores it, which is not typical of something that is going to be used for something. The problem is that FASTCALL procedures (which are regparam(3)) pass arguments in eax, edx, and ecx. So, dumping this constant into edx trashes the second argument to the procedure. A workaround for this bug would seem to be to disable FASTCALL (and I guess that gprof support stopped working when I enabled FASTCALL to fix a different bug).

I released 2.4.7-4um today. The main new thing is that you can change the IP address of a ethertap eth0 device and the host configuration will change to match. This required a bit of infrastructure which I wanted for other reasons. The uml_net interface is now versioned, which I've been meaning to do for a while. uml_net now goes away cleanly when UML is killed messily. Before, it would hang around, occupying the tap device, and when UML was rerun, the new uml_net would emit non-intuitive error messages.

I also made hostfs build and run again on 2.2 with a bit of Makefile hackery.

That hostfs problem turned out to be different than I thought. Livio Soares started chasing the problem and found that the hostfs_user close_file didn't actually close anything. It took a pointer to a file descriptor and closed the pointer (or at least tried to) rather than the descriptor that it pointed to. Fixing that made hostfs behave a lot better.

Having fixed that, I finished the page cache work for UML and it can now successfully do the deb build through hostfs without getting the md5sum mismatches it was getting before. Having said that, I've started seeing a compilation problem when building UML through hostfs that I don't get on the host.

On to OLS. My talk was in the second slot of the first day, which was nice. It's good to get your talk over early so you can do the rest of the conference without worrying about it. It went pretty well. I had hoped to fit a demo in at the end, but the talk basically went the full 90 minutes. So I did a real short watch-it-boot-up demo afterwards while most of the crowd was filing out of the room.

There was a talk on porting Linux to the i-series IBM boxes (aka AS-400) which was fairly interesting. They ported Linux/ppc to a hypervisor running on OS-400, making it fairly similar to the UML port, being a port to an OS rather than to bare hardware. Dave Boucher, who gave the talk, made a number of comments comparing it to UML, which was nice. He also grabbed me during lunch today to quiz me about the COW ubd driver. It turns out that he can't do that so easily because OS-400 doesn't have sparse files, so he can't drop blocks down in the same location in the COW file as in the backing file because that would allocate space. I suggested a block directory instead of a bitmap at the beginning of the COW file and dropping changed blocks down sequentially, but he seemed unconvinced for some reason.

A number of people told me either they or people they knew were using UML for various things. The FreeS/WAN project as a whole seems extremely interested in UML for running tests on their stuff over a virtual network. A PPPoE maintainer complained about the ethertap transport not being intuitively obvious on 2.4. And there were a bunch of other people who were less specific about what their interest in UML was who were either using it or were intending to.

In other news, I discovered that the mcast network transport didn't work when the box had no ethernet card in it. Being at OLS, I showed this to Harald Welte and we stared at the code a bit, then asked Andi Kleen about it. The underlying problem turned out to be that there was no route to any multicast address because there was no interface on the system that supported multicast. The fix seems to be to add multicast support to the loopback device, preferably, and if that's not possible for some reason, to the dummy device.

2.4.7-1um is released. A change which made kernel threads sychronize with the parent at startup caused a hang at boot. The cause was a long-standing bug which caused initdata not to be shared between processes. Andrea noticed the problem as well, and found the fix.

That bug was fixed and I released everything. I released it with a fairly big hostfs problem that I didn't notice until the middle of the release process. I changed how it opens and closes files, with the result that it closes them later than it used to. So, it isn't too hard to get hostfs very confused by running UML out of file descriptors.

2.4.7 appeared yesterday. I'm looking it over to see what's new. One interesting thing is that Alan is sending over some bits of UML which change the generic kernel. These don't affect anything besides UML, so they're harmless. On the other hand, they eliminate some generic files from my patch, which is nice. It makes the UML patch appear purer.

Two of the three patches I sent to Alan were broken again. However, I figured out why. My devious little mail reader was breaking lines when it sent out the mail, which was way too late for me to eyeball it to make sure it wasn't messing up. Turning off this behavior results in much better patches at the other end of the line.

I played with the UML .deb builder and got a workable .deb out of it. I think I figured out why the process gets a checksum error at the very end - it's on hostfs, and hostfs reads through the page cache, but doesn't write through it. I'll have to check with a filesystem guru on this, but it sounds right to me. Putting the process on a normal block device results in good checksums.

Those last two patches made it into ac19. Time to start thinking about bringing the ac tree up to -7um.

I finally got Greg Lonnon's iomem match into UML. This allows a process outside UML to communicate with one inside (or with a UML driver) through a mmapped file.

I've also been chasing the TASK_UNINTERRUPTIBLE hang that a few people have been seeing. It happens most easily under UML, apparently. I'm using a recipe discovered by mistral to reproduce it (two infinite loops each diffing two kernel pools). The longest it's taken to reproduce is about 30 minutes. It hung on boot once. The others have been in the 5-10 minute range.

I had a long chat with Al Viro last night with him telling me what he wanted to see from gdb and me providing it. He ended up being puzzled about what was happening. Following a suggestion from Daniel Phillips, I've started instrumenting buffer_heads and pages to see what happened to the ones involved in the hang.

gdbbot got its first test yesterday when I looked at the problem that Chris Emerson is having with UML/ppc hanging during boot. I didn't find the problem, but was able to check that signal delivery (which was what I thought was broken) was working fine. The next step will be to do a post-mortem on the hang.

I wrote a IRC gateway for gdb. This allows a gdb (like the UML kernel debugger) to be controlled from an IRC channel. The intent is that if someone sees a bug that I can't reproduce, but want to look at, that person's UML gdb can be attached to an IRC channel where I can poke around and see what's going on.

I also integrated Lennert's management console patch. This is a very low-level interface to the kernel (like the i386 SysRq interface). The main use for it right now is to hot-plug devices. At this point, only the ubd driver and gdb support this. So, you can add and remove block devices from your UML without having to reboot it. You can switch gdb in and out the same way. I will also do the consoles, serial lines, and network interfaces at some point as well.

Banged out a bunch of bugs. I started booting UML with 24 megs and plenty of swap, and running a whole bunch of stuff on it to overload it and put it heavily into swap. This turned up a couple of signal delivery races and a swapping bug. The signal races would cause various strange behavior. Mostly what I saw was hangs with an infinite sequence of sigreturns. The swap bug caused pages not to be unmapped when they were swapped out. Obviously, this is very bad. With the help of rcastro, I fiddled my page table macros to fix this. I'm still seeing process segfaults. It looks like pages are being swapped out and swapped back in with the wrong data.

I fixed a bunch of buglets, like the console xterms not going away, readonly hostfs not being readonly, merged Harald Welte's mcast network transport, and a few other things, and checked them into CVS. I also checked in the tools, so everything ought to be up-to-date and consistent at this point.

I also have the .deb build procedure working, I think. The uncertainty is due to the fact that I think there's a hostfs data corruption problem. My development box runs Red Hat, and I couldn't find RPMs for the Debian tools, so I just installed them in my Debian filesystem (apt-get rocks, BTW :-), mount the source pool inside a Debian UML via hostfs, and run the debian build procedure there. The problem is that the gzipped source tarball has its md5sum recorded at the beginning of the build and checked again at the end, and they don't match. I also ran md5sum three times in a row on that file while the builder was running, and got three different answers. So, it looks like I have some debugging to do there.

From the changelog, it looks like yesterday's patches are in ac7. Time to start generating more...

Alan put yesterday's patch into ac5, so UML should build and run again. Thanks to Arjan van de Ven for telling me about that.

I got the networking cleaned up enough that I'm happy for the general public to use it. There are three host transports, ethertap, the routing daemon, and slip. You can have the helper do the host setup for you or not. If you do, then getting the network running is a matter of a command line switch, ifconfiging the device, and setting routes inside UML. This is a huge usability improvement over the previous situation.

This is all checked in, and I'm currently building 2.4.5, which I'll release in the next day or two.

I fixed the slip interface, cleaned out some unused code which had become a portability problem, and fixed the fix for the crash caused by someone typing at the console too soon. It is all checked in to CVS.

I grabbed 2.4.4-ac11 to see if Henrik's patch was in there, and it was. So I don't have to worry about it any more. I guess it made ac9, but Henrik didn't get credit for it in the ac changelog.

In other news, the ethertap interface is working reasonably well. It couldn't do HTTP until I figured out that the mtu on host tap device needed to be 16 bytes less than the UML eth0 mtu. The helper is now more helpful. In order to talk to the rest of the world through it, you basically just have to ifconfig the device inside UML and add a route to the outside world, and you're done. Much better than what we had before.

Five of yesterday's six patches made it into 2.4.4-ac9. The lonely exception was Henrik's hostfs blocksize fix.

The patches I sent in a couple days ago are all in ac10 by the looks of Alan's change log.

I figured out how initrd support is supposed to work, and implemented the necessary stuff in UML. I booted a RH initrd image far enough to convince myself that it works.

I also figured out the sleep hang. It turns out to be a race between the registration of the timer irq and the first time the timer interrupt calls do_IRQ. The timer was enabled before the registration, so if an interrupt happened in that window, do_IRQ would bail out early, leaving the irq permanently marked as in progress and pending. This locked out all future timer interrupts from going through the irq system, so counters would never be decremented, and sleeps would never wake up.

UML is now in 2.4.3-ac4. I was on IRC with Alan and a bunch of other hackers when he merged it. He looked like he was going to start asking a bunch of embarassing questions about my locking, but he was concerned only about one thing, and that was a special case that didn't need locking.

Too bad it doesn't build. The patch that Alan merged was against the Linus 2.4.3 tree, which differs in a few respects from the current -ac tree.

Released 2.4.3 a week or so late. Blame Linus for releasing it the night before the kernel summit officially started. We were all in San Jose and not able to react.

A couple more summit tidbits that I forgot to mention in my last entry:

• Willy is thinking about using UML as a testbed for NUMA support. He wants to fire up a number of virtual machines and have them hook themselves together so they can access each other's memory through device files. This would allow people who don't have access to the fancy hardware to develop and debug Linux support for these boxes.
• UML may appear in the -ac trees at some point. He wanted to include it, but I had sounded fairly negative towards that in the past. What I don't want just yet is for UML to hit the Linus tree. Alan said he doesn't send stuff to Linus if the author doesn't want it sent, which is fine by me.

Back from the kernel summit. I wanted to get a feel for whether four things that I wanted from the host kernel were reasonable. I got two OKs and two dings. That's fine, since the OKs were the important ones. Here's the run-down:

• Userspace manipulation of address spaces : I want to be able to create, populate, release, and switch between mm_structs. This will speed up UML context switches, and greatly clean up that code. I asked Linus, and he said OK to the fairly static things that I want to do. Apparently, there are serious complications when fiddling with the address space of another process, but that's not what I want to do.
• System call interception via signals : In order to avoid the context switching between threads involved in virtualizing a system call, I want to have a process intercept its own system calls by having the host kernel deliver a signal whenever it makes a system call. The handler would be the current syscall_handler, which would read the arguments from its sigcontext_struct. This would change a system call virtualization from four context switches to a signal delivery and return. I infer Alan's OK on this from my describing it in his presence and him not objecting.
• Notification when a UML thread sleeps in the kernel due to a page fault : For the sake of cleanliness and completeness, I want to be able to have UML know when a thread is sleeping in the kernel and be able to call schedule when that happens. This would let UML do as much work as possible given its state of memory residence. Alan rejected this on the grounds that UML would be the only sane user of this mechanism.
• Full kernel preemption : This was implicitly rejected as a UML need by Alan's rejection of the previous item. If UML is to call schedule whenever it sleeps, the whole kernel needs to be preemptible because the swapped-out page might be a kernel page. This doesn't at all mean that preemption isn't going to happen. Rather, it means that UML doesn't have a particular need for it.

Other tidbits:

• A number of people consider UML a very neat hack, including Ben Lahaise, Andrea Arcangeli, and Eric Raymond.
• Alan turns out to be a UML user. For the last month or so, he's been booting his kernels as UML kernels before booting them as native kernels. This is in part because recovery from a totally messed up kernel is a lot easier with UML than with a native kernel. UML is also his ptrace test case. It apparently does things with ptrace that nothing else tries.
• Al Viro is thinking about porting UML to Plan 9. He asked me about what it would take. He had thought through the ptrace requirement, and I told him about the mmap requirement, which is the next hurdle. Plan 9 apparently doesn't have mmap. He's going to think about how to do that.

On the trip over, I did some debugging, and I also threw in some patches. There is now a "umid=<name>" switch for providing a virtual machine with an identifier. This causes a pid file to be created using that name, which is something that makes controlling multiple UMLs through a nice UI a lot easier. This file will be replaced with a socket to the machine console that Lennert is working on.

I also implemented __exitcall, which declares a procedure which is to be called on machine shutdown. This was prompted by the need to remove the pid files when the virtual machine goes away. I also converted other existing cleanups to use this mechanism.

I checked in a bunch of changes again. Henrik Nordstrom provoked me into making it possible to use hostfs as a root directory by sending me a patch that did it, but which was wrong (IMHO). He did it in the same way that nfsroot and initrd support is done, which is by adding a special block of code to fs/super.c inside CONFIG_HOSTFS_ROOT. That works fine, but I didn't want to annoy Al Viro. What I did instead was to add a second registration of hostfs as a device (not a virtual) filesystem and change the ubd driver to support being given a directory rather than a file or block device. What happens is that when a read request comes in to the ubd driver, it is guaranteed to be a request for the superblock. The driver constructs a fake superblock with the directory name in it. hostfs recognizes that and claims the mount as its own. After that, it goes back to being a normal virtual filesystem and doesn't bother the block driver again. The involvement of the ubd driver is a bit of a kludge, but it works well on the command line, and I can't think of anything better besides some kind of general support for virtual root filesystems that cover nfs and initrd as well as hostfs.

There were also a number of patches from other people: Lennert Buytenhek's modify_ldt patch, a bunch from Greg Lonnon, one from Gordon McNutt, and a buffer overrun patch from Henrik Nordstrom.

I spent a few days fixing the infinite recursive context switch bug. That was a lot more complicated than I expected. The fix involved replacing the shadow page tables that represent the mappings on the host for each process with bits in the pte that say whether it is up-to-date or not. These bits are set in the little functions that change ptes and cleared in fix_range after it's updated the mappings. Since the process page tables are per-mm and not per-process, a mapping that was changed in a multi-threaded process would only be updated for one of the threads. This meant that UML processes that share a memory context also need to share a memory context on the host. This in turn complicated exec, since it now needs to create a new host process in order to get out of a shared UML address space. I implemented this a few times, and on about the third try, I got something that works.

So, aside from eliminating a nasty bug, this also makes the modify_ldt fix more useful, since it now should work properly without any extra code, and opens the way to more efficient context switching between threads, since they won't have to go through the remapping that processes need.

Lots of bugs have been fixed. I got a little list of hostfs complaints from Al Viro, which I think I fixed. hostfs is now pretty solid. I fixed the naming problem which cropped up if you held a file open, then moved its directory and accessed that file by its new name. You'd get 'file not found'. This is because I stored the full host pathname in the inode, and when you changed its name while holding it open by the old name, the inode continued to contain the old bogus name. This was fixed by having anything that needs a filename walk the dentry tree back up to the root, constructing the current filename. The other major problem was that readdir didn't work, resulting in missing files when a directory was copied. These are fixed. What remains is to get rid of some interfaces which will complain about not being implemented.

The signal delivery race is fixed. That induced me to clean up a lot of old, crufty code in the kernel entry and exit paths. That's sensitive code, and a few bugs in it caused some very selective and very strange behavior.

I've put together an RPM for UML just in time for the April Linux Magazine to hit the streets with my article in it. This is good, because the article claims that RPMs are available, which they weren't at the time that I wrote it. This also goes some way towards simplifying the network mess. The RPM installs the umn_helper, which lets the umn device run without any help from the user. It also installs the eth tools, which are otherwise hard to find unless you pull them from cvs.

CVS update today. I fixed a few bugs and cleaned up a bunch of things.

I've started keeping an up-to-date TODO list. This will help me not forget anything important. I post it to the -devel list occasionally to prompt people to send in whatever gripes they have.

I'm releasing 2.4.2 today. It has a number of bug fixes and no significant functionality changes.

A number of bugs have cropped up lately. The most significant is a race when a process signal handler returns. There is a narrow window in which an interrupt can cause a crash. The fix is to implement sigreturn like the other arches and run almost all of the kernel code on the kernel stack rather than the process stack as I'm doing now.

I managed to reproduce a number of panics and fixed all but one of them. The key was hitting UML with a high-concurrency ab run with requests that fire off perl scripts which make mySQL requests, with not too much memory, so that it is at least starting to swap.

This reproduced two bugs, one was caused by a failed memory allocation in the middle of setting up a tracing thread request. The failure caused a schedule, which caused a switch request, which blew away the first, partially-set-up request. When the process was rescheduled, its request was garbage, confusing the tracing thread into detaching it. This was fixed by moving the allocation to before the request started being set up.

The other one, which isn't fixed yet, is caused by the shadow page tables maintained by arch/um/kernel/tlb.c. It occasionally needs to allocate a page table when it sets up ptes for a new range of memory. However, if the context switch that it's dealing with was forced by low memory, then that allocation will fail, causing a recursive context switch, and recursion continues until either the stack guard page is hit, or, in the case of a kernel thread, the task structure is polluted. I'm going to fix this by following a suggestion by prumpf, which is to use some spare bits in the pte rather than a separate page table to figure out what parts of the address space need updating.

And, panic number three, which is also fixed, was caused a faulty notion of when a thread is in kernel space. The old way was to look at whether the thread is being traced. That fails when a breapoint was put in a signal handler before it requested that tracing be turned off. The fix is to look at the current stack pointer. However, that causes problems when a signal is being delivered to a process. In this case, there is kernel code running on the process stack. So, a flag was added to the thread structure when this is happening.

Updated the web site with a couple pages describing hostfs and the new console/serial line input specification.

The hostfs memory corruption problems are fixed. slab debug found them for me. They turned out to be two string buffer overrun bugs. I'll release a patch with the fixes pretty soon.

hostfs now pretty much works. I built UML from inside itself on hostfs. I fixed some bugs in the write code, added enough mmap support to run binaries from hostfs, and implemented statfs. However, there is some as-yet explained memory corruption going on.

In an attempt to reproduce the MySQL problems that a couple people are seeing, I moved some of my work, which is heavy on MySQL and perl, into UML. I've seen no problems, which is disappointing because I'm not any closer to finding the bug, but also nice because it shows that it's possible to do real work inside it.

I've also been banging on the ethernet driver trying to reproduce the server buffer overflow that I saw earlier. No dice there either.

The swapoff bug is now fixed. It turned out to be a bad idea to give kernel threads both a non-NULL mm and active_mm. That code has been that way for ages. I have no idea when or why it became a bad thing.

I fixed the known bugs in the block driver. The

dd if=/dev/ubd/0 of=/dev/null

hang was due to the driver returning to the block layer rather than continuing to process the queue when it found an out-of-range I/O request. The dbench corruption was due to the elevator rearranging the request queue while a request was in flight. When that request finished, the interrupt handler was supposed to retire it by removing it from the head of the queue. The problem is that the elevator put some other request at the head, and that request was retired without ever being done. Meanwhile, the original request was pushed back in the queue somewhere, and it got done twice.

Dan Aloni has started the Windows port. He got most of the kernel to compile. There are a number of undefined symbols from files that don't compile yet. Overall, though, it's looking pretty good.

I've got hostfs starting to work reasonably. ls now works, you can cd around and cat things. You can't write anything, create files, or execute them yet.

After a bit of a hiatus, I did a CVS update. A number of buglets relating to running UML as a daemon were fixed. The build was cleaned up - I had hard-coded "gcc" instead of "$(CC)" in my Makefiles, the top-level Makefile is now able to do native and user-mode builds, and I cleaned up the drivers and fs Makefiles so that they let Rules.mk do all the hard work.

I'm also back on hostfs. I fixed the mm problems that it uncovered. It can now do ls on the top-level directory.

Linus finally released the final test10 yesterday, so I made my release last night, with a freshmeat announcement this morning. The stack overflow problems in test9 are fixed by doubling the stack size. There is also an inaccessible page between the two stack pages and the task structure, so there shouldn't be any task structure corruption.

There were a number of other fixes. At the last minute, I found and fixed a nasty race which resulted in the kernel tracing its own system calls, resulting in some nasty stack corruption which made it hard to figure out what happened. UML can now run when its main console is not a terminal (i.e. /dev/null). That didn't work because it flipped the terminal between raw and cooked mode, complaining via printk if the ioctls failed. That led to an infinite recursion of printk error messages which ultimately resulted in a segfault. I also made it possible to mount host devices again. That was broke when I made the block driver check IO requests against the device size so it could report errors for out of bounds IO. It turns out not to be possible to get the size of the media behind a block special file, as far as I can tell. So, as far as the block driver was concerned, block devices had zero size, and all IO was out of bounds.

I also started work on the hostfs filesystem. This is a virtual filesystem which provides access to the host filesystem. The theory is straightforward - vfs calls are converted into the equivalent system calls on the host - but this uncovered a subtle memory management bug. If a libc routine which mallocs memory is called, and the break is increased, that extra memory only exists in that process. If the kernel in another process tries using that memory (or tries calling malloc at all), it will fault. What needs to happen is for the context switching code to see if malloc has increased the size of the data segment and map the new memory into the newly running process. This also raises some SMP issues because when the new memory is mapped in, the other processors will need to be told about it so they can also map it. The same is true of the kernel's virtual memory.

Back from ALS. The talk went pretty well. I'll put the slides up on the site at some point.

I fixed the stack overflow problems that people were seeing. The stack is now two pages long, with an inaccessible third page protecting the task structure, which is on the fourth. Now, any stack overflows will segfault rather than polluting the task structure, making them a lot easier to debug. This is in CVS along with a few other changes.

SGI released a new version of XFS for test5 and I tried to apply it to my test8 um pool, the idea being that I could play with xfs in userspace. The patch went in ok, with some rejects that were not too hard to figure out. After some work, I got it to build. It didn't boot, though. There were some changes in ll_blk_rw.c that I didn't understand, and it looks like they are what resulted in the block device getting a NULL buffer to do I/O into.

So, maybe I'll give XFS another try when SGI gets it slightly more up-to-date.

I found out why the kernel debugging interface doesn't handle breakpoints very well. Setting breakpoints results in process segfaults, floating point exceptions, and other strange behavior. It turns out that do_syscall stored the current register state in the thread structure while determining whether the process was doing a system call. If the process hit a breakpoint in the kernel instead, then that overwrote the state that was stored when the system call was called. When the system call returned, that bogus state was restored, and the process was essentially teleported back into the kernel just after the breakpoint, leading to all kinds of strange behavior. With that problem fixed, things work much better. The kernel debugger seems to be basically healthy, and works just like on a normal process.

While I was fixing breakpoints, I decided to see why gdb inside a virtual machine crashes it whenever it sets a breakpoint. There turn out to be a number of problems. First, SIGTRAP wasn't being delivered to the debuggee when it hit a breakpoint. This made it hard for gdb to find out that the breakpoint had been hit, and to remove it temporarily so the debuggee could get by it. Then, it turned out that PTRACE_SINGLESTEP wasn't implemented. This is used by gdb to execute the instruction which had the breakpoint and stop on the next one. There were one or two other buglets, but now that they are fixed, gdb seems happy with breakpoints.

So, I've been a little lax. Here's what's happened in the last few weeks: two bugs were fixed, the reboot bug and and shell segfault bug. That's it.

Finished my ALS paper and sent it in. That's a load off my mind. Made some more CVS checkins. This makes the various debugging options configurable, although I haven't tested the gprof and gcov configurations. I also added some compatibility code to make the Debian install happier. It now recognizes that uml can have disks, but the disk recognizer gets stuck in state 'D' for reasons I haven't figured out yet.

Linus just released test7, so I am building it right now (right now as I'm writing this, and not right now as you're reading it, because I've got no idea when you're reading this). A quick check of the patch shows no changes that I need to worry about, so this looks like a drop-it-in-and-it-just-works patch. We'll see...

Things look good. I booted it up, and ran a few things, and they all worked. So, I'll run the stress testers on it tomorrow, and if that checks out, I'll release it.

Fixed a network driver bug which caused a crash when ab was run against it. This might also fix the ping flood problem. The fix is in cvs. It will appear for real in test7.

I found out what was causing uml not to boot. It turned out to be a casting bug which was making the compiler do pointer arithmetic rather than integer arithmetic. This was a long-standing bug, and test6 changed things so it got hit more heavily. So, assuming that it's not too badly broken now, I'll release test6 for real.

Revamped the website. It will be put up as soon as Linus releases test6 and I've integrated the changes in. This is because this site talks about stuff which isn't really going to be released until then.

Checked in changes which make the new debugging interface more or less work. I also added a 'debug' command line switch which starts the kernel in the debugger, so you have control of it from the start.

There are some problems with it. Commands attached to breakpoints cause segfaults for some reason. It also can't step across a context switch.

I also put in Rusty's patches. They completely revamp the config mechanism. For some reason, there also seems to be very complete networking/netfilter converage.

Got a couple of patches from Rusty. I'm apparently going to graduate to a complete port once I've applied one of them :-) It is nice. It gives me what looks like a complete config process rather than the one I've kludged together. He also sent in enough exports to allow his stuff to be modular inside a UML.

I'm integrating in Lars Brinkhoff's ptrace proxy. It's partially working - enough that I can attach to the running thread, poke around it, set breakpoints, etc. This without needing to detach it from the uml tracing thread. I can't ^C gdb and have it stop the kernel wherever it happens to be. It also doesn't seem to be following threads as one goes to sleep and another starts up. Once these work, this will be a huge improvement in uml kernel debugging.

Linus released test5 last night, so I'm putting out the user-mode version today. There's nothing new in it. The virtual ethernet is in the patch, but not enabled in the binary kernels and off by default in defconfig.

The stress testing of this kernel produced no strange happenings. Maybe the segfaults and other stuff in the last release weren't my fault (heh).

Started integrating Jim Leu's virtual ethernet driver. It basically works, but it misbehaves a fair bit. It's unclear whether that's the kernel's fault or the driver's.

Announced test4 on freshmeat today.

Also discovered a few more problems which I didn't see on test2 or test3:

the occasional process segfault which I've mentioned already

a devfs segfault - I did this by displaying an xterm out to the host; when I logged out, the kernel paniced with memory corruption in devfs

X clients sometimes can't display against a local X server - strace says that they're stuck in select

strace also displayed their read masks as '[?]', which doesn't seem right

Patches for these will be forthcoming when I find fixes.