File: debugging_slow_builds.md

package info (click to toggle)
chromium 139.0.7258.127-1
  • links: PTS, VCS
  • area: main
  • in suites:
  • size: 6,122,068 kB
  • sloc: cpp: 35,100,771; ansic: 7,163,530; javascript: 4,103,002; python: 1,436,920; asm: 946,517; xml: 746,709; pascal: 187,653; perl: 88,691; sh: 88,436; objc: 79,953; sql: 51,488; cs: 44,583; fortran: 24,137; makefile: 22,147; tcl: 15,277; php: 13,980; yacc: 8,984; ruby: 7,485; awk: 3,720; lisp: 3,096; lex: 1,327; ada: 727; jsp: 228; sed: 36
file content (62 lines) | stat: -rw-r--r-- 2,502 bytes parent folder | download | duplicates (11)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# Debugging Slow Builds

Did you know that Ninja writes a log to disk after each build?

To see what kinds of files took the longest for your previous build:

```sh
cd out/Default
# Lives in depot_tools:
post_build_ninja_summary.py
```

Because the build is highly parallelized the `elapsed time` values are usually
not meaningful so the `weighted time` numbers are calculated to approximate
the impact of build steps on wall-clock time.

You can also set `NINJA_SUMMARIZE_BUILD=1` to have this command run
after each `autoninja` invocation. Setting this environment variable also runs
ninja with `-d stats` which causes it to print out internal information such
as StartEdge times, which measures the times to create processes, and it
modifies the `NINJA_STATUS` environment variable to add information such as how
many processes are running at any given time - both are useful for detecting
slow process creation. You can get this last benefit on its own by setting
`NINJA_STATUS=[%r processes, %f/%t @ %o/s : %es ] ` (trailing space is
intentional).

To generate a Chrome trace of your most recent build:

```sh
git clone https://github.com/nico/ninjatracing
ninjatracing/ninjatracing out/Default/.ninja_log > trace.json
# Then open in https://ui.perfetto.dev/
```

If your build is stuck on a long-running build step you can see what it is by
running `tools/buildstate.py`.

## Slow Bot Builds

Our bots run `ninjatracing` and `post_build_ninja_summary.py` as well.

Find the trace at: `postprocess for reclient > gsutil upload ninja_log > ninja_log`:

 * _".ninja_log in table format (full)"_ is for `post_build_ninja_summary.py`.
 * _"trace viewer (sort_by_end)"_ is for `ninjatracing`.

## Advanced(ish) Tips

* Use `gn gen --tracelog trace.json` to create a trace for `gn gen`.
* Many Android templates make use of
  [`md5_check.py`](https://cs.chromium.org/chromium/src/build/android/gyp/util/md5_check.py)
  to optimize incremental builds.
  * Set `PRINT_BUILD_EXPLANATIONS=1` to have these commands log which inputs
    changed.
* If you suspect files are being rebuilt unnecessarily during incremental
  builds:
  * Use `ninja -n -d explain` to figure out why ninja thinks a target is dirty.
  * Ensure actions are taking advantage of ninja's `restat=1` feature by not
    updating timestamps on outputs when their contents do not change.
    * E.g. by using [`build_utils.AtomicOutput()`]

[`build_utils.AtomicOutput()`]: https://source.chromium.org/search?q=symbol:AtomicOutput%20f:build