1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
|
Using perf
==========
Top::
sudo perf top -p `pidof ceph-osd`
To capture some data with call graphs::
sudo perf record -p `pidof ceph-osd` -F 99 --call-graph dwarf -- sleep 60
To view by caller (where you can see what each top function calls)::
sudo perf report --call-graph caller
To view by callee (where you can see who calls each top function)::
sudo perf report --call-graph callee
:note: If the caller/callee views look the same you may be
suffering from a kernel bug; upgrade to 4.8 or later.
Common Issues
-------------
Ceph use `RelWithDebInfo` as its default `CMAKE_BUILD_TYPE`. Hence `-O2 -g` is
used to compile the tree in this case. And the `-O2` optimization level
enables `-fomit-frame-pointer` by default. But this prevents stack profilers
from accessing the complete stack information. So one can disable this option
when launching `cmake` ::
cmake -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer"
or when building the tree::
make CMAKE_CXX_FLAGS="-fno-omit-frame-pointer"
Flamegraphs
-----------
First, get things set up::
cd ~/src
git clone https://github.com/brendangregg/FlameGraph
Run ceph, then record some perf data::
sudo perf record -p `pidof ceph-osd` -F 99 --call-graph dwarf -- sleep 60
Then generate the flamegraph::
sudo perf script | ~/src/FlameGraph/stackcollapse-perf.pl > /tmp/folded
~/src/FlameGraph/flamegraph.pl /tmp/folded > /tmp/perf.svg
firefox /tmp/perf.svg
|
