There were no structured benchmarks of icecream, so I started some. I'm benchmarking 5 runs
each, throwing away the worst and the best run, and then averaging the rest of the 3.
There are two machines in the cluster, both single cpu and both about the same speed (1.7GHz
Pentium M). Normally the tests are done via otherwise idle WLAN (54MBit). WLAN has a bad
latency and very bad throughput, which gives icecream a hard time, and should be compareable
to a loaded cabled network environment. For comparison, I repeated some tests via 100MBit
cabled LAN.

I'm testing linux 2.6.16 (defconfig), which is a set of small C files with sometimes larger
C files inbetween. its a tough benchmark because compiling C is rather quick, so remote
overhead has to be low.

No icecream:

make  -j1:     315s
make  -j2:     322s
make  -j3:     324s
make -j10:     334s

result: without icecream, starting more compilers than CPUs available is a bad idea.


icecream wrapper, no scheduler.

make  -j1:     323s
make -j10:     340s

result: Overhead of just using icecream without cluster is neglectible (2%) in all cases.

dirk's no-latency icecream: remote daemon with -m 1:

make  -j1:     418s
make  -j2:     397s
make  -j3:     263s
make -j10:     230s
make -j10/100: 203s
make -j100:    231s
make -j10/100: 202s

result: Enabling icecream without parallel compilation is a horrible mistake. icecream
must be tuned to detect and compensate this situation better. Maximum performance
improvement of icecream is 27% (LAN: 36%) out of the theoretical 50%.

======================================================================

Qt 3.3.6's src/ subdirectory. This is a bunch of medium and large C++ files. It
gives icecream good opportunities because compilation time is comparatively low
and the preprocessed files are not too large (low STL usage).


No icecream:

make  -j1:     368s
make  -j3:     363s

result: apparently there is a small win by starting more compilers than CPUs in parallel.
Perhaps the I/O overhead is not neglectible like in the linux kernel case.

dirk's no-latency icecream, remote daemon with -m 2:

make  -j1:     572s
make  -j3:     273s
make -j10:     269s
make -j10/100: 239s
make -j100:    282s

result: again, not compiling in parallel is a deadly sin. trying to overload the cluster
with very high parallelism as well. Maximum performance improvement is again 27% and
36% for LAN. That these numbers compare equally with the Linux kernel case is astonishing
and needs explanation.

Now, to make sure that the no-latency patch hasn't regressed anything, we're comparing with
stock 0.7 (which already has some performance improvements over 0.6):


make  -j1:      569s
make -j10:      349s
make -j10/100:  253s
make -j100/100: 278s

It is remarkable, that 0.7 does not provide much advantage over compiling locally (6% speedup)
in a WLAN, while providing the expected 36% speedup for LAN. This proves that no-latency
provides significant wins for unstable/bad network connections and does not regress
performance for good networking setups. The overall 20% improvement is not bad at all.

2006-06-16 make-it-cool-branch:

make -j10/100:  244s
make -j1:       376s

result: correcting process accounting for local jobs makes -j1 fast again (just 2% overhead)

icecream, always compile remote even though host is not faster:

make -j10:       538s
make -j10/sched: 389s

As we can see, the scheduler improves performance by 38%.


======================================================================

New performance experiments. 

New baseline:

make-it-cool, both with -m 1, make -j5

make -j5  :    382s
make -j5  :    354s
make -j5  :    336s
make -j5  :    355s

remote with -m 2

make -j5  :    333s
make -j5  :    299s
make -j5  :    307s

remote with -m 2, preload scheduler:

make -j5  :    303s
make -j5  :    287s
make -j5  :    291s

remote with -m 1, preload scheduler:

make -j5  :    287s
make -j5  :    288s
make -j5  :    289s

remote with -m 1, preload scheduler, optimized return:
make -j5  :    288s
make -j5  :    289s
make -j5  :    288s

remote with -m 2, preload scheduler, optimized return:
make -j5  :    279s
make -j5  :    281s
make -j5  :    278s

As we can see, over-booking the remote slave improves performance by 13%.
As the CPU hardly gets faster, it means that we're reducing idle wait time
this way. 

One experiment was to pre-load jobs on the remote slave. This means even 
though all compile slots are filled, it gets one (exactly one) job assigned.
The daemon itself will start compilation as soon as it gets a free slot,
reducing both client<->CS and CS<->scheduler roundtrip. Overall, it gave an
impressive 4% speedup.

A lot of time is however spent on writing back the compiled object file
to the client, and this is dominated by network saturation and latency
and not by CPU usage. The obvious solution is to notify the scheduler
about a free slot as soon as compilation (but not write-back) has
finished. With remote over-booking this resulted in another 3% speedup,
while no improvements could be measured in the -m 1 case. Given that
it significantly reduced code complexity in the daemon, it should still be
an improvement (reduced code size by almost 8%!).

======================================================================

Load detection tunings.

The biggest problem with accurate load detection is that it is impossible
to find out how much cpu time a iceccd child is currently using. All you
can get is how much time it used overall, but only when it exited. 
Which gives you a lot of cpu-usage peaks sprinkled over time, and you have
to somehow average that out in a meaningful way. 

Actually, the Linux kernel tracks cpu time, and you can read it in
/proc/<pid>/stat for any child. unfortunately it is converted to seconds
in there, so resolution isn't much more accurate. 

Watching the old load detector, it became obvious that it once in a 
while jumps to 1000 because of own jobs eating 100% cpu, but not finishing
within the measure timeframe, which causes the host to be blacklisted
by the scheduler, even though nothing is wrong with it. 

There are two solutions: 

- trying to get a more accurate usage over time

- only track usage whenever it is accurate, e.g. a child exited. 

As the second possibility has problems with multi-cpu environments (you'd
have to wait for all jobs to finish before doing another idleness sample,
which probably reduces throughput considerably), first one was chosen. 

Simplest idea: assume that overall system nice load is at least partially
caused by our own childs. 

remote with -m 2:
make -j5  :    272s
make -j5  :    274s
make -j5  :    271s

Hmm.. 2% win.

============================================================================

New baseline: 0.8.0:

remote with -m 1:
make -j5  :  257s

without compression: 

make -j5:   442s