1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
|
Debugging Firefox with Pernosco
===============================
This page is intended to help Firefox/Gecko developers get started using `Pernosco <https://pernos.co/>`_
to debug Firefox.
Mozilla employees have a paid subscription to Pernosco, and are encouraged to use it.
Contributors can use Pernosco as well. `A free account <https://pernos.co/pricing/>`_
comes with 5 uploads to see if it is suitable for your workflow.
Prerequisites
-------------
- You need the ability to produce an :code:`rr` trace. See :doc:`debugging_firefox_with_rr`
- Employees, you can also generate traces for Pernosco for some CI job failures via
Treeherder.
Workflow
--------
At a high level, working with Pernosco consists of recording a trace then uploading
the trace for processing.
Install your Pernosco configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using your own Pernoso account, follow the instructions
`here <https://pernos.co/account>`_ to install your credentials.
Mozilla employees, see the `Pernosco install and configuration doc
<https://docs.google.com/document/d/1cb-mkNyJYMudpS0a-R6zxy0kOMIRUSKrxxL-E9zFGKE/edit>`_
for credentials.
In general, you will want:
* Your email in :code:`~/.config/pernosco/user`
* Your group in :code:`~/.config/pernosco/group`
* Your key in :code:`~/.config/pernosco/user_secret_key`
Gather a Local Trace
~~~~~~~~~~~~~~~~~~~~
In general, to gather a trace follow the instruction for recording :code:`rr` traces
in :doc:`debugging_firefox_with_rr`. However, to gather a local rr trace that’s
compatible with Pernosco you need to disable some incompatible CPU features (currently
only AVX-512)
On an Intel machine, you can disable this at the command line. To simplify things a
shell script :code:`pernosco-record` can be created like so,
.. code:: bash
#!/bin/bash
rr record --disable-avx-512 "$@"
replacing plain :code:`rr` in command lines (:code:`mach --debugger pernosco-record`)
On AMD Zen Machines, you can disable AVX-512 and 5 level page tables at boot.
On a Debian derived distro this involves editing :code:`/etc/default/grub` and
adding :code:`clearcpuid=304,no5lvl` to :code:`GRUB_CMDLINE_LINUX_DEFAULT`, then
running :code:`sudo update-grub` after.
Upload the trace
~~~~~~~~~~~~~~~~
Find the trace you’re interested in :code:`~/.local/share/rr/<trace>`, and call :code:`pernosco-submit`
.. code:: bash
pernosco-submit upload ~/.local/share/rr/<trace> PATH_TO_CENTRAL --title "Name Of Trace"
where :code:`PATH TO CENTRAL` is the path to the source of your build, checked out
to the revision matching the build.
The last trace recorded is always symlinked, so you can use :code:`~/.local/share/rr/latest-trace` too.
Wait for email
~~~~~~~~~~~~~~
After the trace is processed, you'll get an email with a link to your trace. Good
hunting!
Treeherder Workflow
-------------------
You can also try to get traces generated via the Treeherder interface. On a failing
test that is eligible, in "Artifacts and Debugging Tools", you'll see a link to
.. image:: ../img/record-this-failure.png
:width: 35 %
This link brings you to the self-service reproducer:
.. image:: ../img/self-service.png
:width: 60 %
Select a test and a count, and whether or not chaos-mode should be used, then click
reproduce. At this point there is a **long** wait; the page will notify you when
reproduction has started with an :code:`alert` dialog.
|
