File: rr.html

package info (click to toggle)
rr 2.0.0-1
links: PTS, VCS
area: main
in suites: jessie, jessie-kfreebsd
size: 2,376 kB
ctags: 3,373
sloc: cpp: 17,766; ansic: 5,798; python: 604; asm: 312; sh: 308; makefile: 10
file content (1467 lines) | stat: -rw-r--r-- 41,846 bytes
<!DOCTYPE html>

<meta charset="utf-8">
<title>rr</title>

<section>
  <h1>How <a href="http://rr-project.org">rr</a> works</h1>
  <footer><a href="www.mozilla.org/research/">mozilla
      research</a></footer>
</section>

<section>
  <p>rr records nondeterministic executions and debugs them
    deterministically.
  <p>Practical tool; version 1.2 is latest release.  Used to debug
    Firefox.
</section>

<!-----------------------------------------------------------------------------
-- Why?
-->
<section>
  <h2>Why?</h2>
</section>

<section>
  <p>Deterministic debugging: record nondeterministic failure once,
    debug deterministically forever.
</section>

<section>
  <p>Record intermittent test failures "at scale" online, debug the
    recordings offline at leisure.
</section>

<section>
  <p>Omniscient debugging: issue queries over program state changes;
    go backwards in time.
</section>

<!-----------------------------------------------------------------------------
-- Overview
-->
<section>
  <h2>Overview</h2>
</section>

<section>
  <p><code>rr record prog --args</code><br />
    &rarr;<em>saves recording</em>
  <p><code>rr replay</code><br /> &rarr;<em>debugger socket drives
    replay of most recent recording</em>
</section>

<section>
  <p>Most of an application's execution is deterministic.
  <p>rr records the <em>nondeterministic</em> parts.
</section>

<section>
  <h3>Examples of nondeterministic inputs</h3>
  <ul>
    <li><code>clock_gettime(...<span class="highlight">&now</span>);</code>
    <li><code>read(fd, <span class="highlight">buf</span>,
        4096);</code>
    <li><code>__asm__("<span class="highlight">rdtsc</span>")</code>
    <li><code>ioctl(<span class="highlight">...</span>)</code>
    <li>UNIX signals...
  </ul>
</section>

<section>
  <p>Then during replay, emulate system calls and <code>rdtsc</code>
  by writing the saved nondeterministic data back to the tracee.
</section>

<section>
  <p>Shared-memory multitasking is a nondeterministic "input".
  <p>... but modern hardware can't record it efficiently.  So rr
  doesn't record truly parallel executions.
</section>

<section>
  <h3>Scheduling tasks</h3>
  <p>Can switch tasks at syscalls.  Must preempt straight-line code
  too; <em>and replay the preemptions deterministically</em>.
</section>

<section>
  <h3>Hardware performance counters (HPCs)</h3>
  <p>Recent chips count instructions-retired, branches-retired,
  ..., and can be programmed to interrupt after a count
  of <i>x</i>.
</section>

<section>
  <p>Simulate task preemption with HPC interrupts.
  <p>Idea: program insns-retired counter to interrupt
  after <i>k</i> <sup>&#10033;</sup>.  That <i>k</i> approximates
  a <em>time slice</em>.
</section>

<section>
  <h3>Replaying preemption</h3>
  <p>Record the insn-retired counter value <i>v</i> to the trace file.
  During replay, program the interrupt for <i>v</i>.  Voilà.
</section>

<section>
  <p>UNIX signals are recorded and replayed like task preemptions.
  <p>Record counter value <i>v</i> and signum.  Replay by interrupting
  after <i>v</i> and "delivering" signum.
</section>

<!-----------------------------------------------------------------------------
-- System requirements
-->
<section>
  <h2>System requirements</h2>
</section>

<section>
  <h3>Basic requirements</h3>
  <ul>
    <li>Intel chip with Nehalem (2010) or later µarch<sup>1</sup>. VM
      with perf counter virtualization is OK.
    <li>x86 userspace. x86-64 kernel is OK.
    <li>linux with <code>PTRACE_INTERRUPT</code> support: &ge;
      3.4
    <li>(<i>strongly encouraged</i>) linux with seccomp-bpf support:
      &ge; 3.5
  </ul>
  <footer><sup>1</sup> Some Haswell chips don't work with rr
    (yet).</footer>
</section>

<section>
  <p>rr touches low-level details of machine architecture, by
    necessity; f.e. kernel syscall ABI.
  <p>Supporting more ISAs is "just work". x86-64 coming.
</section>

<section>
  <p>ARM chips don't have the performance counters that rr requires.
  <p>So no ARM support is possible at the moment.
</section>

<section>
  <p>Precise HPC events identify points in execution.
  <p>Precise replay of signals and preemption requires interrupting
  tracees at these events.
</section>

<section>
  <h3><sup>&#10033;</sup>Performance counters are messier in reality</h3>
  <ul>
    <li>Insns-retired counter is imprecise.  <em>Use precise
      retired-branch counter instead</em>.
    <li>Counter interrupts can overshoot.  <em>Subtract a "skid
      region"</em>.
    <li>(<em>So replay point is technically indeterminate.  But
      doesn't seem to be a problem in practice, yet</em>.)
  </ul>
</section>

<section>
  <p>seccomp-bpf enables rr to <em>selectively trace</em>
  syscalls.
  <p>Only trap to rr for syscalls that can't be handled in the tracee.
  Over 100x faster in µbenchmarks.
</section>

<section>
  <h3>Buffer syscalls; flush buffer as "super event"</h3>
  <p><img style="width: 100%;" src="res/syscall-buffer.svg">
</section>

<!-----------------------------------------------------------------------------
-- Recorder implementation
-->
<section>
  <h2>Recorder implementation</h2>
</section>

<section>
  <p>Tasks are controlled through the ptrace API.
  <p>HPCs are controlled through the perf event API.
</section>

<section>
  <p>The first traced task is forked from rr.  After
  that, <code>clone()</code> and <code>fork()</code>from tracees
  add new tasks.
  <p>And tasks die at <code>exit()</code>.
</section>

<section>
  <h3>Simplified recorder loop</h3>
  <!-- TODO use incremental list?  ol seems to be broken. -->
  <pre>
    while live_task():
        task t = <span class="highlight">schedule()</span>
        if not <span class="highlight">status_changed(t)</span>:
            <span class="highlight">resume_execution(t)</span>
        <span class="highlight">state_change(t)</span>
    </pre>
    <footer>src/recorder.cc</footer>
</section>

<section>
  <h3>Scheduling a task</h3>
  <pre>
    task <span class="highlight">schedule()</span>:
        for each task t, round-robin:
            if is_runnable(t)
               or <span class="highlight">status_changed(t)</span>:
                return t
        tid = waitpid(ANY_CHILD_TASK)
        return task_map[tid]
  </pre>
  <footer>src/recorder_sched.cc</footer>
</section>

<section>
  <h3>Tasks changing status</h3>
  <pre>
    bool <span class="highlight">status_changed(task t)</span>:
        # Non-blocking
        return waitpid(t.tid, WNOHANG)

    <em># Deceptively simple: includes
    # syscalls, signals, ptrace
    # events ...</em>
  </pre>
  <footer>src/task.cc</footer>
</section>

<section>
  <h3>Resuming task execution</h3>
  <p><em>Invariant</em>: At most one task is running userspace code.
  All other tasks are either idle or awaiting completion of a
  syscall.<sup>&#8224;</sup>
</section>

<section>
  <p>Multiple running tasks suffer from <em>shared-memory
      hazards</em>.
  <p>rr doesn't attempt to record these hazards, so can't replay them
    deterministically.
</section>

<section>
  <h3>Resuming a task, simplified</h3>
    <pre>
    void <span class="highlight">resume_execution(task t)</span>:
        ptrace(PTRACE_SYSCALL, t.tid)
        waitpid(t.tid)  # Blocking

    <em># Again, deceptively simple: traps
    # for syscalls, signals, ptrace
    # events ...</em>
  </pre>
  <footer>src/{recorder, task}.cc</footer>
</section>

<section>
  <p>Most recorder work is done
  for <code><span class="highlight">state_change(task
  t)</span></code>.
  <p>But before looking at it, a few digressions ...
</section>

<section>
  <h3>Generating time-slice interrupts</h3>
  <ul>
    <li><code>perf_event_open()</code> fd for
      retired-conditional-branches; details are µarch
      specific
    <li>Set event "sample period" to <i>k</i>
    <li>Make event fd O_ASYNC and set tracee task as owner
    <li>&rarr; <em>tracee sent SIGSTKFLT at rbc ≈ <i>k</i></em>
  </ul>
  <footer>src/hpc.cc</footer>
</section>

<section>
  <h3>Trapping tracees at rdtsc</h3>
  <ul>
    <li><code>prctl(PR_SET_TSC, PR_TSC_SIGSEGV)</code> &rarr; tracees
      executing rdtsc trap to SIGSEGV
    <li>rr examines which instruction triggered SIGSEGV
    <li>if rdtsc, value is recorded by rr tracer and tracee insn is
      emulated
  </ul>
</section>

<section>
  <p>Tracees generate ptrace events by executing fork, clone, exit,
    and some other syscalls.
  <p>ptrace events exist for linux reasons that aren't
  interesting.
</section>

<section>
  <p>(rr tracees can share memory mappings with other processes.
  <p>Not possible to record efficiently in SW; needs kernel and/or HW
  support. Unsupported until then.)
</section>

<section>
  <h3>Tracee events recorded
    by <code><span class="highlight">state_change()</span></code></h3>
  <ul>
    <li>"Pseudo"-signals delivered by implementation of rdtsc or
      time-slice interrupts
    <li>Other, "real", signals
    <li>ptrace events
    <li>Syscall entry and exit
  </ul>
</section>

<section>
  <p>Some syscalls must be executed atomically; can't switch task
  until syscall finishes.
  <p>Ex: <code>mmap</code> modifies address space, can race other
    syscalls.
</section>

<section>
  <p>On the other hand, some syscalls <em>require</em> switching;
  syscall can't finish until the task switches.
  <p>Ex: <code>waitpid()</code> only returns after child runs, changes
    state.
</section>

<section>
  <p><em>Problem</em>: kernel writes non-atomic syscall outparams in
    an order that rr can't record.
  <p>Kernel outparam writes race rr tracees in userspace, syscalls.
</section>

<section>
  <p>Solution: allocate <em>scratch space</em> for the outparams of
    non-atomic syscalls.  At syscall exit, write scratch data back to
    outparams.
  <p><em>&rarr; rr orders outparam writes</em>
</section>

<section>
  <p>POSIX signals can arrive at practically any point in execution
    and invoke signal handler code.
  <p><em>&rarr; tracee code can be (almost) arbitrarily
      re-entered</em>
</section>

<section>
  <p>Linux exits tracees out of syscalls with
    an <code>ERESTART*</code> error code before delivering signals.
    Syscall not always restarted after signal.
  <p>Sighandler
      nesting <a href="https://github.com/mozilla/rr/wiki/Linux-signals">gets
      complex</a>.
</section>

<section>
  <h3>When a signal becomes pending</h3>
  <ul>
    <li>consult sighandler table to see if there's a registered
      sighandler function
    <li>if so, <code>SINGLESTEP</code> into the sighandler frame and
      record the <code>struct sigframe</code> set up by kernel. Also
      record sighandler registers.
    <li>otherwise, deliver the signal using the ptrace API
</section>

<section>
  <p>Sighandlers exit using the <code>SYS_sigreturn</code> syscall.
    rr uses these calls to help determine whether interrupted syscalls
    are restarted.
</section>

<section>
  <p>Tracees exit in unpredictable order at fatal signals like
  SIGABRT. Naïve <code>waitpid()</code> calls deadlock.
  <p><code>exit_group</code>: same problem.
</section>

<section>
  <h3>"Unstable" tracee exit</h3>
  <p>rr solves this by detaching and not waiting on affected tracees.
  <p><sup>&#8224;</sup>Breaks rr scheduling invariant.
</section>

<!-----------------------------------------------------------------------------
-- Syscall buffer
-->
<section>
  <h2>Syscall buffer</h2>
</section>

<section>
  <p>ptrace traps are expensive.  Better to do as much work in tracee
    process as possible.
  <p>Use seccomp-bpf to selectively trap syscalls.
</section>

<section>
  <p>Syscall hooks are <code>LD_PRELOAD</code>'d into tracees.
  <p>Hook functions record kernel return value and outparam data to
    the <em>syscall buffer</em>.
</section>

<section>
  <p>rr monkeypatches <code>__kernel_vsyscall()</code> in vdso to jump
    to rr trampoline.
  <p>Trampoline calls dispatcher, which calls rr hook if
    available.
</section>

<section>
  <p><em>Untraced</em> syscalls are recorded to syscallbuf by tracee.
  <em>Traced</em> events recorded by the rr process "flush" the
  tracee's syscallbuf.
  <p>Lib falls back on traced syscalls.
</section>

<section>
  <h3>Simplified example of syscallbuf hook function</h3>
  <pre style="margin: 50px 50px; font-size: 24px">
static int sys_close(int fd)
{
	long ret;
	if (!start_buffer_syscall(SYS_close))
		/* Fall back on traced syscall.
                 * This generates a ptrace trap. */
		return syscall(SYS_close, fd);

	/* Untraced syscall. Does not generate
         * ptrace trap.*/
	ret = untraced_syscall1(SYS_close, fd);
        /* Save the result to syscall buffer. */
	return commit_syscall(SYS_close, ret);
}
  </pre>
</section>

<section>
  <h3>How untraced syscalls are made</h3>
  <ul>
    <li>Create single "untraced" kernel entry point
      <pre>
asm("_untraced_syscall:\n\t"
    "int $0x80");
      </pre>
    <li>Install seccomp-bpf filter that passes calls
    from <code>_untraced_syscall</code>, traps to rr otherwise.
    </ul>
</section>

<section>
  <p>seccomp-bpf traps generate <code>PTRACE_EVENT_SECCOMP</code> in
    tracer process.
  <p>rr can then <code>PTRACE_SYSCALL</code> the tracee into traced
    syscall.
</section>

<section>
  <p><em>Problem</em>: buffered syscalls don't trap to rr, by design.
    But may-block syscalls (f.e. <code>waitpid()</code>) require rr to
    schedule another task.
</section>

<section>
  <h3>perf events to the rescue: "descheduled" event</h3>
  <p>Set event to fire on tracee context switch.  Event traps to rr.
  <p><em>Buffered syscall blocks &rarr; context switch &rarr; rr
      trap</em>
</section>

<section>
  <h3>Generating desched events</h3>
  <ul>
    <li>In <em>tracee</em>, <code>perf_event_open()</code> fd for
      context-switch counter
    <li>Set event "sample period" to <code>1</code> (i.e. next context
      switch) just before buffered syscall
    <li>Disarm event just after buffered syscall.
    <li>&rarr; <em>tracee sent SIGSYS if context switched during
        buffered syscall</em>
  </ul>
  <footer>src/preload/preload.c</footer>
</section>

<!-----------------------------------------------------------------------------
-- Saved traces
-->
<section>
  <h2>Saved traces</h2>
</section>

<section>
  <p>Traces are saved to <code>~/.rr</code> by default.
  <p>Stored on disk uncompressed. Trace compression is planned.
</section>

<section>
  <h3>Trace directory contents</h3>
  <ul>
    <li><code>args_env</code>: command-line args and environment
      pairs used to <code>execvpe()</code> initial tracee
    <li><code>events</code>: sequence of syscalls, signals, and
      various other execution events
    <li><code>mmaps</code>: metadata about mmap'd files
    <li><code>data</code>/<code>data_header</code>: all recorded data,
      along with metadata about when it was recorded
    <li><code>version</code>: trace format version number
  </ul>
</section>

<!-----------------------------------------------------------------------------
-- Replayer implementation
-->
<section>
  <h2>Replayer implementation</h2>
</section>

<section>
  <p>Emulate most syscalls using trace data.
  <p>Actually execute a small number.
</section>

<section>
  <h3>Built around PTRACE_SYSEMU</h3>
  <p><code>SYSCALL</code> runs tracee to syscall, executes it.
  <p><code>SYSEMU</code> runs to syscall, <em>doesn't</em> execute it.
    rr replays side effects.
</section>

<section>
  <h3>Replaying time-slice interrupts, in theory</h3>
  <p>Program instructions counter to interrupt after the
    recorded <i>t</i> number of instructions.
  <p>Tracee stops at <i>t</i>.
</section>

<section>
  <h3>Replaying time-slice interrupts, in practice</h3>
  <ul>
    <li>have to use retired-conditional-branch counter (RBC)
    <li>RBC interrupts aren't precise.  Can overshoot by up to 70
      branches.
    <li>RBC counter value doesn't uniquely identify a point in
      execution (unlike retired-insn counter value)
  </ul>
</section>

<section>
  <h3>Finding execution target, in practice</h3>
  <ul>
    <li>program RBC interrupt for <i>target-rbc - SKID_SIZE</i>
    <li>after RBC interrupt, set breakpoint on target <code>$ip</code>
      to avoid single-stepping when possible
    <li>when breakpoint hit, compare RBC value and register files to
      guess if at execution target.  If RBC and regs match what was
      recorded, done.
  </ul>
  <p>To reiterate, this is not sound.
</section>

<section>
  <p>Deterministic signals were raised by program execution.  For
    example, <code>*NULL = 42;</code>
  <p>Replayed "naturally" in the course of execution.
</section>

<section>
  <p>Async signals were raised externally at some execution point
    during recording.
  <p>Replay to that execution point <em>just as for time-slice
    interrupts</em>.
</section>

<section>
  <h3>Replay signal delivery by emulating</h3>
  <p>If there was a sighandler, restore recorded <code>sigframe</code>
    and registers at sighandler entry.
  <p>Otherwise, nothing else to do.
</section>
<section>
  <h3>Replaying buffered syscalls</h3>
  <p>Read saved buffer from trace.
  <p>Replay each syscall as normal, but restore outparam data from
    records in the read buffer.
</section>

<!-----------------------------------------------------------------------------
-- Debugger interface
-->
<section>
  <h2>Debugger interface</h2>
</section>

<section>
  <p>Common commands supported.
  <p><code>c, s, si, b, bt, watch, info regs, thr, info thr ...</code>
</section>

<section>
  <p><code>(gdb) call foo()</code> can cause replay divergence.
  <p>So you're not allowed to do it &hellip; for now. Support coming.
</section>

<section>
  <p>Small stub translates from and to gdb remote protocol.
  <p>Then passes debugger requests up to rr replayer.
  <footer>src/debugger_gdb.cc</footer>
</section>

<section>
  <p>Replayer fulfills requests using <code>ptrace()</code> or cached
  data.
  <p>And resumes tracee execution when asked.
  <footer>src/replayer.cc</footer>
</section>

<section>
  <p>Breakpoints, <code>int $3</code>, <code>stepi</code>, watchpoints
    all raise <code>SIGTRAP</code>.
  <p><code>$ip</code>, breakpoint table, gdb request,
    and <code>$DR6</code> decode trap.
  <footer>src/replayer.cc</footer>
</section>

<!-----------------------------------------------------------------------------
-- Future work
-->
<section>
  <h2>Future work</h2>
</section>

<section>
  <h3><a href="https://github.com/mozilla/rr/wiki/Checkpointing-(deepfork)-during-replay">Checkpointing</a></h3>
  <p>Make a "deep fork" of tracee tree during replay.
  <p>Run code (or whatever) in copied tree, return to original.
</section>

<section>
  <h3><a href="https://github.com/mozilla/rr/wiki/Chroniclerr">Omniscient
      debugging (aka chroniclerr)</a></h3>
  <p>Use <a href="https://code.google.com/p/chronicle-recorder/">chronicle</a>-style
    instrumentation to generate execution DB.
  <p>Query state changes in DB.
</section>

<section>
  <h3><a href="http://research.microsoft.com/en-us/projects/chess/">CHESS</a>-style
    execution search; targeted recording</h3>
  <p>At each scheduling decision, make a checkpoint.
  <p>If execution reaches bad state, done.  Else, resume checkpoint.
</section>

<section>
  <h3>Other projects</h3>
  <ul>
    <li>Copy traces across machines
    <li>Integrate hardware shared-memory multithreading recorder
      like <a href="http://iacoma.cs.uiuc.edu/iacoma-papers/isca13_1.pdf">QuickRec</a>
    <li>Record ptrace API; <code>rr record rr record</code>&hellip;
    <li>Handle GPU drivers (NVIDIA, ATI, ...)
    <li>Port to Darwin kernel
    <li>Port to Windows NT kernel
    <li><s>ARM port not possible with current generation of chips</s>
  </ul>
</section>

<section>
  <h3>Thanks from the rr team!</h3>
  <ul>
    <li><a href="http://rr-project.org/">rr-project.org</a>
    <li><a href="https://github.com/mozilla/rr">github.com/mozilla/rr</a>
    <li><a href="https://mail.mozilla.org/listinfo/rr-dev">mail.mozilla.org/listinfo/rr-dev</a>
    <li>#research on irc.mozilla.org
  </ul>
</section>

<section>
  <h2>Appendix: rr for RnR people</h2>
</section>

<section>
  <p>Release 1.2 available today at
  <p><a href="http://rr-project.org/">rr-project.org</a>
</section>

<section>
  <h3>Use cases</h3>
  <ul>
    <li>Run on modern, commodity hardware and software: <code>yum
    install rr; rr record</code>&hellip;
    <li>Aspire to general tool, focus on Firefox initially
    <li>Record nondeterministic test failures at scale (e.g., Firefox
    build/test infra), debug offline
    <li>"Super-debugger" for local development
    <li>Search execution space to actively find bugs
  </ul>
</section>

<section>
  <h3>Design concerns</h3>
  <ul>
    <li>Commodity HW &rarr; only record single HW thread (for
      now!)
    <li>Commodity SW &rarr; stick to higher-level userspace APIs
      (e.g., ptrace, PEBS)
    <li>Record tests at scale &rarr; record perf must be "economical",
      but not mission-critical
    <li>"Super-debugger" &rarr; the usual, plus queries over execution
      history; pretty fast replay
    <li>Search exe space &rarr; flexible scheduling and
      checkpointing
</section>

<section>
  <h3>rr recorder overview</h3>
  <ul>
    <li>Record "applications" consisting of linux tasks
    <li>Schedule CPU slices by programming precise counter interrupt
      (retired branches, RBC) for <i>k</i>
    <li>Time slice is special case of signal: time-slice recorded as
      <i>(0, rec-RBC)</i>, signals <i>(signum, rec-RBC)</i>
    <li>Record kernel-created signal stack
    <li>Syscall "outparams" and <code>rdtsc</code> generate trace
      traps; results saved to log
    <li><i>Plus a "faster" mode we'll cover later</i>
</section>

<section>
  <h3>Trade-off: scheduling from userspace</h3>
  <ul>
    <li>While it's great to fully control scheduling ...
    <li>... we have to approximate timeslices; can be unfair
    <li>... interactive programs don't "feel" native; rr has its own
      heuristics
    <li>... can be slower
    <li>Future work is to have the option of both
  </ul>
</section>

<section>
  <h3>Headache: kernel writes racing with userspace</h3>
  <ul>
    <li>rr doesn't (can't efficiently) record reads/writes of memory
      shared outside of tracee application.
    <li>But there are still hazards with the kernel:
    <li>... kernel-write/task-read hazard on syscall outparam buffers
      &rarr; rr replaces user buffers with scratch and serializes
      writes
    <li>... kernel-write/task-read on random futexes,
      e.g. CLONE_CHILD_CLEARTID &rarr; no good solution yet;
      usleep&hellip;
  </ul>
</section>

<section>
  <h3>rr replayer overview</h3>
  <ul>
    <li>Replay signal/time-slice <i>(signum, rec-RBC)</i> by
      programming interrupt for <i>rec-RBC</i>
    <li>Emulate signals by restoring signal stack and regs
    <li>Emulate syscalls by restoring outparams where possible
    <li>Execute non-emulatable clone/mmap/et al. as required
    <li>Serve debugger requests (maybe covered later)
</section>

<section>
  <h3>Replayer headache: slack in counter interrupts</h3>
  <ul>
    <li>Interrupt programmed for <i>RBC = k</i> may actually fire at
      up to <i>RBC = k + slack</i>
    <li>(Slack empirically seen to be >= 70 branches)
    <li>So we program interrupt for <i>RBC = k - slack</i> and then
      advance by breakpoint+stepi
    <li>"At target" when <i>rec-RBC == rep-RBC</i> and <i>[rec-regs]
        == [rep-regs]</i>
    <li>Replay target therefore technically indeterminate
  </ul>
</section>

<section>
  <h3>Recorder "fast mode": syscall buffering</h3>
  <ul>
    <li>ptrace traps are slow
    <li><em>Idea</em>: avoid them when possible by buffering log data
      in tracee task
    <li>Implementation: LD_PRELOAD a helper library with hooks for
      common syscalls (read, write, gettimeofday, etc.)
    <li>Hook makes fast untraced syscall, saves outparams in
      task-local buffer
    <li>Flush buffer at traced event (including buffer overflow)
</section>

<section>
  <h3>Headache: many syscalls made internally in glibc</h3>
  <ul>
    <li>Those syscalls can't be wrapped by usual approach of
    interposing exported symbol using LD_PRELOAD
    <li>Solution: monkeypatch <code>__kernel_vsyscall()</code> in
      vdso.
    <li>Syscalls directly made through <code>int $0x80</code> still
      can't be buffered.
    <li>We hope this terrible hack evolves into kernel support.
</section>

<section>
  <h3>Headache: buffering syscalls that may block</h3>
  <ul>
    <li>read/write/&hellip; may block if buffer empty/full/&hellip;
    <li>But, <em>untraced</em> syscall from wrapper means no trap to
      rr for scheduling arbitration
    <li>If another tracee is blocked too, then may deadlock
    <li>Solution: libc wrapper programs perf_event interrupt triggered
      by next context-switch of task
    <li>If the syscall blocks, task is switched out, and rr tracer
      gets interrupt (SIGIO from perf_event)
  </ul>
</section>

<section>
  <h3>Fun debugging tricks</h3>
  <ul>
    <li>Save register / RBC info at all events, verify in replay
    <li>Generate memory checksums at selected events, verify in
      replay
    <li>LLVM pass to add "execution path logging" (Bell-Larus): poor
      man's log of retired branches.  Save to "magic fd" in recording,
      verify in replay.
    <li>Hack replayer itself to efficiently log arbitrary info at
      arbitrary points
  </ul>
</section>

<style>
.highlight {
  color: red;
  background-color: yellow;
  font-weight: bold;
}
.todo {
  color: aqua;
  background-color: purple;
  font-weight: bold;
}
</style>

<!-- Maybe a font from http://www.google.com/webfonts ? -->
<link href='http://fonts.googleapis.com/css?family=Oswald' rel='stylesheet'>

<style>
  html, .view body { background-color: black; counter-reset: slideidx; }
  body, .view section { background-color: white; border-radius: 12px }
  /* A section is a slide. It's size is 800x600, and this will never change */
  section, .view head > title {
      /* The font from Google */
      font-family: 'Oswald', arial, serif;
      font-size: 30px;
  }

  .view section:after {
    counter-increment: slideidx;
    content: counter(slideidx, decimal-leading-zero);
    position: absolute; bottom: -80px; right: 100px;
    color: white;
  }

  .view head > title {
    color: white;
    text-align: center;
    margin: 1em 0 1em 0;
  }

  h1, h2 {
    margin-top: 200px;
    text-align: center;
    font-size: 80px;
  }
  h3 {
    margin: 50px 0 25px 50px;
  }

  ul {
      margin: 25px 100px;
  }
  li > ul {
      margin: 15px 50px;
  }

  p {
    margin: 75px;
    font-size: 50px;
  }

  blockquote {
    height: 100%;
    background-color: black;
    color: white;
    font-size: 60px;
    padding: 50px;
  }
  blockquote:before {
    content: open-quote;
  }
  blockquote:after {
    content: close-quote;
  }

  /* Figures are displayed full-page, with the caption
     on top of the image/video */
  figure {
    background-color: black;
    width: 100%;
    height: 100%;
  }
  figure > * {
    position: absolute;
  }
  figure > img, figure > video {
    width: 100%; height: 100%;
  }
  figcaption {
    margin: 70px;
    font-size: 50px;
  }

  footer {
    position: absolute;
    bottom: 0;
    width: 100%;
    padding: 40px;
    text-align: right;
    background-color: #F3F4F8;
    border-top: 1px solid #CCC;
  }

  /* Transition effect */
  /* Feel free to change the transition effect for original
     animations. See here:
     https://developer.mozilla.org/en/CSS/CSS_transitions
     How to use CSS3 Transitions: */
  section {
    -moz-transition: left 400ms linear 0s;
    -webkit-transition: left 400ms linear 0s;
    -ms-transition: left 400ms linear 0s;
    transition: left 400ms linear 0s;
  }
  .view section {
    -moz-transition: none;
    -webkit-transition: none;
    -ms-transition: none;
    transition: none;
  }

  .view section[aria-selected] {
    border: 5px red solid;
  }

  /* Before */
  section { left: -150%; }
  /* Now */
  section[aria-selected] { left: 0; }
  /* After */
  section[aria-selected] ~ section { left: +150%; }

  /* Incremental elements */

  /* By default, visible */
  .incremental > * { opacity: 1; }

  /* The current item */
  .incremental > *[aria-selected] { opacity: 1; }

  /* The items to-be-selected */
  .incremental > *[aria-selected] ~ * { opacity: 0; }

  /* The progressbar, at the bottom of the slides, show the global
     progress of the presentation. */
  #progress-bar {
    height: 2px;
    background: #AAA;
  }
</style>

<!-- {{{{ dzslides core
#
#
#     __  __  __       .  __   ___  __
#    |  \  / /__` |    | |  \ |__  /__`
#    |__/ /_ .__/ |___ | |__/ |___ .__/ core :€
#
#
# The following block of code is not supposed to be edited.
# But if you want to change the behavior of these slides,
# feel free to hack it!
#
-->

<div id="progress-bar"></div>

<!-- Default Style -->
<style>
  * { margin: 0; padding: 0; -moz-box-sizing: border-box; -webkit-box-sizing: border-box; box-sizing: border-box; }
  [role="note"] { display: none; }
  body {
    width: 800px; height: 600px;
    margin-left: -400px; margin-top: -300px;
    position: absolute; top: 50%; left: 50%;
    overflow: hidden;
    display: none;
  }
  .view body {
    position: static;
    margin: 0; padding: 0;
    width: 100%; height: 100%;
    display: inline-block;
    overflow: visible; overflow-x: hidden;
    /* undo Dz.onresize */
    transform: none !important;
    -moz-transform: none !important;
    -webkit-transform: none !important;
    -o-transform: none !important;
    -ms-transform: none !important;
  }
  .view head, .view head > title { display: block }
  section {
    position: absolute;
    pointer-events: none;
    width: 100%; height: 100%;
  }
  .view section {
    pointer-events: auto;
    position: static;
    width: 800px; height: 600px;
    margin: -150px -200px;
    float: left;

    transform: scale(.4);
    -moz-transform: scale(.4);
    -webkit-transform: scale(.4);
    -o-transform: scale(.4);
    -ms-transform: scale(.4);
  }
  .view section > * { pointer-events: none; }
  section[aria-selected] { pointer-events: auto; }
  html { overflow: hidden; }
  html.view { overflow: visible; }
  body.loaded { display: block; }
  .incremental {visibility: hidden; }
  .incremental[active] {visibility: visible; }
  #progress-bar{
    bottom: 0;
    position: absolute;
    -moz-transition: width 400ms linear 0s;
    -webkit-transition: width 400ms linear 0s;
    -ms-transition: width 400ms linear 0s;
    transition: width 400ms linear 0s;
  }
  .view #progress-bar {
    display: none;
  }
</style>

<script>
  var Dz = {
    remoteWindows: [],
    idx: -1,
    step: 0,
    html: null,
    slides: null,
    progressBar : null,
    params: {
      autoplay: "1"
    }
  };

  Dz.init = function() {
    document.body.className = "loaded";
    this.slides = Array.prototype.slice.call($$("body > section"));
    this.progressBar = $("#progress-bar");
    this.html = document.body.parentNode;
    this.setupParams();
    this.onhashchange();
    this.setupTouchEvents();
    this.onresize();
    this.setupView();
  }

  Dz.setupParams = function() {
    var p = window.location.search.substr(1).split('&');
    p.forEach(function(e, i, a) {
      var keyVal = e.split('=');
      Dz.params[keyVal[0]] = decodeURIComponent(keyVal[1]);
    });
  // Specific params handling
    if (!+this.params.autoplay)
      $$.forEach($$("video"), function(v){ v.controls = true });
  }

  Dz.onkeydown = function(aEvent) {
    // Don't intercept keyboard shortcuts
    if (aEvent.altKey
      || aEvent.ctrlKey
      || aEvent.metaKey
      || aEvent.shiftKey) {
      return;
    }
    if ( aEvent.keyCode == 37 // left arrow
      || aEvent.keyCode == 38 // up arrow
      || aEvent.keyCode == 33 // page up
    ) {
      aEvent.preventDefault();
      this.back();
    }
    if ( aEvent.keyCode == 39 // right arrow
      || aEvent.keyCode == 40 // down arrow
      || aEvent.keyCode == 34 // page down
    ) {
      aEvent.preventDefault();
      this.forward();
    }
    if (aEvent.keyCode == 35) { // end
      aEvent.preventDefault();
      this.goEnd();
    }
    if (aEvent.keyCode == 36) { // home
      aEvent.preventDefault();
      this.goStart();
    }
    if (aEvent.keyCode == 32) { // space
      aEvent.preventDefault();
      this.toggleContent();
    }
    if (aEvent.keyCode == 70) { // f
      aEvent.preventDefault();
      this.goFullscreen();
    }
    if (aEvent.keyCode == 79) { // o
      aEvent.preventDefault();
      this.toggleView();
    }
  }

  /* Touch Events */

  Dz.setupTouchEvents = function() {
    var orgX, newX;
    var tracking = false;

    var db = document.body;
    db.addEventListener("touchstart", start.bind(this), false);
    db.addEventListener("touchmove", move.bind(this), false);

    function start(aEvent) {
      aEvent.preventDefault();
      tracking = true;
      orgX = aEvent.changedTouches[0].pageX;
    }

    function move(aEvent) {
      if (!tracking) return;
      newX = aEvent.changedTouches[0].pageX;
      if (orgX - newX > 100) {
        tracking = false;
        this.forward();
      } else {
        if (orgX - newX < -100) {
          tracking = false;
          this.back();
        }
      }
    }
  }

  Dz.setupView = function() {
    document.body.addEventListener("click", function ( e ) {
      if (!Dz.html.classList.contains("view")) return;
      if (!e.target || e.target.nodeName != "SECTION") return;

      Dz.html.classList.remove("view");
      Dz.setCursor(Dz.slides.indexOf(e.target) + 1);
    }, false);
  }

  /* Adapt the size of the slides to the window */

  Dz.onresize = function() {
    var db = document.body;
    var sx = db.clientWidth / window.innerWidth;
    var sy = db.clientHeight / window.innerHeight;
    var transform = "scale(" + (1/Math.max(sx, sy)) + ")";

    db.style.MozTransform = transform;
    db.style.WebkitTransform = transform;
    db.style.OTransform = transform;
    db.style.msTransform = transform;
    db.style.transform = transform;
  }


  Dz.getNotes = function(aIdx) {
    var s = $("section:nth-of-type(" + aIdx + ")");
    var d = s.$("[role='note']");
    return d ? d.innerHTML : "";
  }

  Dz.onmessage = function(aEvent) {
    var argv = aEvent.data.split(" "), argc = argv.length;
    argv.forEach(function(e, i, a) { a[i] = decodeURIComponent(e) });
    var win = aEvent.source;
    if (argv[0] === "REGISTER" && argc === 1) {
      this.remoteWindows.push(win);
      this.postMsg(win, "REGISTERED", document.title, this.slides.length);
      this.postMsg(win, "CURSOR", this.idx + "." + this.step);
      return;
    }
    if (argv[0] === "BACK" && argc === 1)
      this.back();
    if (argv[0] === "FORWARD" && argc === 1)
      this.forward();
    if (argv[0] === "START" && argc === 1)
      this.goStart();
    if (argv[0] === "END" && argc === 1)
      this.goEnd();
    if (argv[0] === "TOGGLE_CONTENT" && argc === 1)
      this.toggleContent();
    if (argv[0] === "SET_CURSOR" && argc === 2)
      window.location.hash = "#" + argv[1];
    if (argv[0] === "GET_CURSOR" && argc === 1)
      this.postMsg(win, "CURSOR", this.idx + "." + this.step);
    if (argv[0] === "GET_NOTES" && argc === 1)
      this.postMsg(win, "NOTES", this.getNotes(this.idx));
  }

  Dz.toggleContent = function() {
    // If a Video is present in this new slide, play it.
    // If a Video is present in the previous slide, stop it.
    var s = $("section[aria-selected]");
    if (s) {
      var video = s.$("video");
      if (video) {
        if (video.ended || video.paused) {
          video.play();
        } else {
          video.pause();
        }
      }
    }
  }

  Dz.setCursor = function(aIdx, aStep) {
    // If the user change the slide number in the URL bar, jump
    // to this slide.
    aStep = (aStep != 0 && typeof aStep !== "undefined") ? "." + aStep : ".0";
    window.location.hash = "#" + aIdx + aStep;
  }

  Dz.onhashchange = function() {
    var cursor = window.location.hash.split("#"),
        newidx = 1,
        newstep = 0;
    if (cursor.length == 2) {
      newidx = ~~cursor[1].split(".")[0];
      newstep = ~~cursor[1].split(".")[1];
      if (newstep > Dz.slides[newidx - 1].$$('.incremental > *').length) {
        newstep = 0;
        newidx++;
      }
    }
    this.setProgress(newidx, newstep);
    if (newidx != this.idx) {
      this.setSlide(newidx);
    }
    if (newstep != this.step) {
      this.setIncremental(newstep);
    }
    for (var i = 0; i < this.remoteWindows.length; i++) {
      this.postMsg(this.remoteWindows[i], "CURSOR", this.idx + "." + this.step);
    }
  }

  Dz.back = function() {
    if (this.idx == 1 && this.step == 0) {
      return;
    }
    if (this.step == 0) {
      this.setCursor(this.idx - 1,
                     this.slides[this.idx - 2].$$('.incremental > *').length);
    } else {
      this.setCursor(this.idx, this.step - 1);
    }
  }

  Dz.forward = function() {
    if (this.idx >= this.slides.length &&
        this.step >= this.slides[this.idx - 1].$$('.incremental > *').length) {
        return;
    }
    if (this.step >= this.slides[this.idx - 1].$$('.incremental > *').length) {
      this.setCursor(this.idx + 1, 0);
    } else {
      this.setCursor(this.idx, this.step + 1);
    }
  }

  Dz.goStart = function() {
    this.setCursor(1, 0);
  }

  Dz.goEnd = function() {
    var lastIdx = this.slides.length;
    var lastStep = this.slides[lastIdx - 1].$$('.incremental > *').length;
    this.setCursor(lastIdx, lastStep);
  }

  Dz.toggleView = function() {
    this.html.classList.toggle("view");

    if (this.html.classList.contains("view")) {
      $("section[aria-selected]").scrollIntoView(true);
    }
  }

  Dz.setSlide = function(aIdx) {
    this.idx = aIdx;
    var old = $("section[aria-selected]");
    var next = $("section:nth-of-type("+ this.idx +")");
    if (old) {
      old.removeAttribute("aria-selected");
      var video = old.$("video");
      if (video) {
        video.pause();
      }
    }
    if (next) {
      next.setAttribute("aria-selected", "true");
      if (this.html.classList.contains("view")) {
        next.scrollIntoView();
      }
      var video = next.$("video");
      if (video && !!+this.params.autoplay) {
        video.play();
      }
    } else {
      // That should not happen
      this.idx = -1;
      // console.warn("Slide doesn't exist.");
    }
  }

  Dz.setIncremental = function(aStep) {
    this.step = aStep;
    var old = this.slides[this.idx - 1].$('.incremental > *[aria-selected]');
    if (old) {
      old.removeAttribute('aria-selected');
    }
    var incrementals = $$('.incremental');
    if (this.step <= 0) {
      $$.forEach(incrementals, function(aNode) {
        aNode.removeAttribute('active');
      });
      return;
    }
    var next = this.slides[this.idx - 1].$$('.incremental > *')[this.step - 1];
    if (next) {
      next.setAttribute('aria-selected', true);
      next.parentNode.setAttribute('active', true);
      var found = false;
      $$.forEach(incrementals, function(aNode) {
        if (aNode != next.parentNode)
          if (found)
            aNode.removeAttribute('active');
          else
            aNode.setAttribute('active', true);
        else
          found = true;
      });
    } else {
      setCursor(this.idx, 0);
    }
    return next;
  }

  Dz.goFullscreen = function() {
    var html = $('html'),
        requestFullscreen = html.requestFullscreen || html.requestFullScreen || html.mozRequestFullScreen || html.webkitRequestFullScreen;
    if (requestFullscreen) {
      requestFullscreen.apply(html);
    }
  }
  
  Dz.setProgress = function(aIdx, aStep) {
    var slide = $("section:nth-of-type("+ aIdx +")");
    if (!slide)
      return;
    var steps = slide.$$('.incremental > *').length + 1,
        slideSize = 100 / (this.slides.length - 1),
        stepSize = slideSize / steps;
    this.progressBar.style.width = ((aIdx - 1) * slideSize + aStep * stepSize) + '%';
  }
  
  Dz.postMsg = function(aWin, aMsg) { // [arg0, [arg1...]]
    aMsg = [aMsg];
    for (var i = 2; i < arguments.length; i++)
      aMsg.push(encodeURIComponent(arguments[i]));
    aWin.postMessage(aMsg.join(" "), "*");
  }
  
  function init() {
    Dz.init();
    window.onkeydown = Dz.onkeydown.bind(Dz);
    window.onresize = Dz.onresize.bind(Dz);
    window.onhashchange = Dz.onhashchange.bind(Dz);
    window.onmessage = Dz.onmessage.bind(Dz);
  }

  window.onload = init;
</script>


<script> // Helpers
  if (!Function.prototype.bind) {
    Function.prototype.bind = function (oThis) {

      // closest thing possible to the ECMAScript 5 internal IsCallable
      // function 
      if (typeof this !== "function")
      throw new TypeError(
        "Function.prototype.bind - what is trying to be fBound is not callable"
      );

      var aArgs = Array.prototype.slice.call(arguments, 1),
          fToBind = this,
          fNOP = function () {},
          fBound = function () {
            return fToBind.apply( this instanceof fNOP ? this : oThis || window,
                   aArgs.concat(Array.prototype.slice.call(arguments)));
          };

      fNOP.prototype = this.prototype;
      fBound.prototype = new fNOP();

      return fBound;
    };
  }

  var $ = (HTMLElement.prototype.$ = function(aQuery) {
    return this.querySelector(aQuery);
  }).bind(document);

  var $$ = (HTMLElement.prototype.$$ = function(aQuery) {
    return this.querySelectorAll(aQuery);
  }).bind(document);

  $$.forEach = function(nodeList, fun) {
    Array.prototype.forEach.call(nodeList, fun);
  }

</script>
<!-- vim: set fdm=marker: }}} -->