1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
# Scala library benchmarks
This directory is a standalone SBT project, within the Scala project,
that makes use of the [SBT plugin](https://github.com/ktoso/sbt-jmh) for [JMH](http://openjdk.java.net/projects/code-tools/jmh/).
## Running a benchmark
The benchmarks require first building Scala into `../../build/pack` with `ant`.
If you want to build with `sbt dist/mkPack` instead,
you'll need to change `scalaHome` in this project.
You'll then need to know the fully-qualified name of the benchmark runner class.
The benchmarking classes are organized under `src/main/scala`,
in the same package hierarchy as the classes that they test.
Assuming that we're benchmarking `scala.collection.mutable.OpenHashMap`,
the benchmark runner would likely be named `scala.collection.mutable.OpenHashMapRunner`.
Using this example, one would simply run
jmh:runMain scala.collection.mutable.OpenHashMapRunner
in SBT.
SBT should be run _from this directory_.
The JMH results can be found under `target/jmh-results/`.
`target` gets deleted on an SBT `clean`,
so you should copy these files out of `target` if you wish to preserve them.
## Creating a benchmark and runner
The benchmarking classes use the same package hierarchy as the classes that they test
in order to make it easy to expose, in package scope, members of the class under test,
should that be necessary for benchmarking.
There are two types of classes in the source directory:
those suffixed `Benchmark` and those suffixed `Runner`.
The former are benchmarks that can be run directly using `jmh:run`;
however, they are normally run from a corresponding class of the latter type,
which is run using `jmh:runMain` (as described above).
This …`Runner` class is useful for setting appropriate JMH command options,
and for processing the JMH results into files that can be read by other tools, such as Gnuplot.
The `benchmark.JmhRunner` trait should be woven into any runner class, for the standard behavior that it provides.
This includes creating output files in a subdirectory of `target/jmh-results`
derived from the fully-qualified package name of the `Runner` class.
## Some useful HotSpot options
Adding these to the `jmh:run` or `jmh:runMain` command line may help if you're using the HotSpot (Oracle, OpenJDK) compiler.
They require prefixing with `-jvmArgs`.
See [the Java documentation](http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html) for more options.
### Viewing JIT compilation events
Adding `-XX:+PrintCompilation` shows when Java methods are being compiled or deoptimized.
At the most basic level,
these messages will tell you whether the code that you're measuring is still being tuned,
so that you know whether you're running enough warm-up iterations.
See [Kris Mok's notes](https://gist.github.com/rednaxelafx/1165804#file-notes-md) to interpret the output in detail.
### Consider GC events
If you're not explicitly performing `System.gc()` calls outside of your benchmarking code,
you should add the JVM option `-verbose:gc` to understand the effect that GCs may be having on your tests.
### "Diagnostic" options
These require the `-XX:+UnlockDiagnosticVMOptions` JVM option.
#### Viewing inlining events
Add `-XX:+PrintInlining`.
#### Viewing the disassembled code
If you're running OpenJDK or Oracle JVM,
you may need to install the disassembler library (`hsdis-amd64.so` for the `amd64` architecture).
In Debian, this is available in
<a href="https://packages.debian.org/search?keywords=libhsdis0-fcml">the `libhsdis0-fcml` package</a>.
For an Oracle (or other compatible) JVM not set up by your distribution,
you may also need to copy or link the disassembler library
to the `jre/lib/`_`architecture`_ directory inside your JVM installation directory.
To show the assembly code corresponding to the code generated by the JIT compiler for specific methods,
add `-XX:CompileCommand=print,scala.collection.mutable.OpenHashMap::*`,
for example, to show all of the methods in the `scala.collection.mutable.OpenHashMap` class.
To show it for _all_ methods, add `-XX:+PrintAssembly`.
(This is usually excessive.)
## Useful reading
* [OpenJDK advice on microbenchmarks](https://wiki.openjdk.java.net/display/HotSpot/MicroBenchmarks)
* Brian Goetz's "Java theory and practice" articles:
* "[Dynamic compilation and performance measurement](http://www.ibm.com/developerworks/java/library/j-jtp12214/)"
* "[Anatomy of a flawed benchmark](http://www.ibm.com/developerworks/java/library/j-jtp02225/)"
* [Doug Lea's JSR 166 benchmarks](http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/test/loops/)
* "[Measuring performance](http://docs.scala-lang.org/overviews/parallel-collections/performance.html)" of Scala parallel collections
## Legacy frameworks
An older version of the benchmarking framework is still present in this directory, in the following locations:
<dl>
<dt><code>bench</code></dt>
<dd>A script to run the old benchmarks.</dd>
<dt><code>source.list</code></dt>
<dd>A temporary file used by <code>bench</code>.</dd>
<dt><code>src/scala/</code></dt>
<dd>The older benchmarks, including the previous framework.</dd>
</dl>
Another, older set of benchmarks is present in `../benchmarking/`.
|
