1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
|
---
layout: documentation
title: Creating persistent workers
---
# Creating persistent workers
[Persistent workers](persistent-workers.html) can make your build faster.
If you have repeated actions in your build that have a high startup cost or
would benefit from cross-action caching, you may want to implement your own
persistent worker to perform these actions.
The worker implementation has two parts:
* The [worker](#making-the-worker),
* The [rule that uses the worker](#making-the-rule-that-uses-the-worker).
## Making the worker
A worker upholds a few requirements:
* It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/6d1b9725b1e201ca3f25d8ec2a730a20aab62c6e/src/main/protobuf/worker_protocol.proto#L35)
from its `stdin`.
* It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/6d1b9725b1e201ca3f25d8ec2a730a20aab62c6e/src/main/protobuf/worker_protocol.proto#L49)
(and only `WorkResponse`s) to its `stdout`.
* It accepts the `--persistent_worker` flag.
If your program upholds these requirements, it can be used as a worker!
### Work requests
A `WorkRequest` contains a list of arguments to the worker, a list of path-digest
pairs representing the inputs the worker can access (this isn’t enforced, but
you can use this info for caching), and a request id, which is 0 for singleplex
workers.
```json
{
“args” : [“--some_argument”],
“inputs” : [
{ “/path/to/my/file/1” : “fdk3e2ml23d”},
{ “/path/to/my/file/2” : “1fwqd4qdd” }
],
“request_id” : 12
}
```
### Work responses
A `WorkResponse` should contain the same request id, a zero or nonzero exit
code, and an output string that contains any errors encountered in processing
or executing the request. Workers may write additional output to `stderr`, but
they must only write `WorkResponse`s to `stdout`.
```json
{
“exit_code” : 1,
“output” : “Action failed with the following message:\nCould not find input
file “/path/to/my/file/1”,
“request_id” : 12
}
```
As per the norm for protobufs, the fields are optional. However, Bazel requires
the `WorkRequest` and the corresponding `WorkResponse`, to have the same request
id, so the request id must be specified if it is nonzero. This is a valid
`WorkResponse`.
```json
{
“request_id” : 12,
}
```
A `request_id` of 0 indicates a "singleplex" request, i.e. this request cannot
be processed in parallel with other requests. The server guarantees that a
given worker receives requests with either only `request_id` 0 or only
`request_id` greater than zero. Singleplex requests are sent in serial, i.e. the
server doesn't send another request until it has received a response (except
for cancel requests, see below).
**Notes**
* Each protocol buffer is preceded by its length in `varint` format (see
[`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-).
* JSON requests and responses are not preceded by a size indicator.
* JSON requests uphold the same structure as the protobuf, but use standard
JSON.
* Bazel stores requests as protobufs and converts them to JSON using
[protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java)
### Cancellation
Workers can optionally allow work requests to be cancelled before they finish.
This is particularly useful in connection with dynamic execution, where local
execution can regularly be interrupted by a faster remote execution. To allow
cancellation, add `supports-worker-cancellation: 1` to the
`execution-requirements` field (see below) and set the
`--experimental_worker_cancellation` flag.
A **cancel request** is a `WorkRequest` with the `cancel` field set (and
similarly a **cancel response** is a `WorkResponse` with the `was_cancelled`
field set). The only other field that must be in a cancel request or cancel
response is `request_id`, indicating which
request to cancel. The `request_id` field will be 0 for singleplex workers
or the non-0 `request_id` of a previously sent `WorkRequest` for multiplex
workers. The server may send cancel requests for requests that the worker has
already responded to, in which case the cancel request must be ignored.
Each non-cancel `WorkRequest` message must be answered exactly once, whether
or not it was cancelled. Once the server has sent a cancel request, the worker
may respond with a `WorkResponse` with the `request_id` set
and the `was_cancelled` field set to true. Sending a regular `WorkResponse`
is also accepted, but the `output` and `exit_code` fields will be ignored.
Once a response has been sent for a `WorkRequest`, the worker must not touch
the files in its working directory. The server is free to clean up the files,
including temporary files.
## Making the rule that uses the worker
You'll also need to create a rule that generates actions to be performed by the
worker. Making a Starlark rule that uses a worker is just like [creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules).
In addition, the rule needs to contain a reference to the worker itself, and
there are some requirements for the actions it produces.
### Referring to the worker
The rule that uses the worker needs to contain a field that refers to the worker itself,
so you'll need to create an instance of a `\*\_binary` rule to define your
worker. If your worker is called `MyWorker.Java`, this might be the associated
rule:
```python
java_binary(
name = “worker”,
srcs = [“MyWorker.Java”],
)
```
This creates the "worker" label, which refers to the worker binary. You'll then
define a rule that *uses* the worker. This rule should define an attribute that
refers to the worker binary.
If the worker binary you built is in a package named "work", which is at the top
level of the build, this might be the attribute definition:
```python
"worker": attr.label(
default = Label("//work:worker"),
executable = True,
cfg = "host",
)
```
`cfg = "host"` indicates that the worker should be built to run on your host
platform.
### Work action requirements
The rule that uses the worker creates actions for the worker to perform. These
actions have a couple of requirements.
* The _“arguments”_ field. This takes a list of strings, all but the last
of which are arguments passed to the worker upon startup. The last element in
the “arguments” list is a `flag-file` (@-preceded) argument. Workers read
the arguments from the specified flagfile on a per-WorkRequest basis. Your
rule can write non-startup arguments for the worker to this flagfile.
* The _“execution-requirements”_ field, which takes a dictionary containing
`“supports-workers” : “1”`, `“supports-multiplex-workers” : “1”`, or both.
The "arguments" and "execution-requirements" fields are required for all
actions sent to workers. Additionally, actions that should be executed by
JSON workers need to include `“requires-worker-protocol” : “json”` in the
execution requirements field. `“requires-worker-protocol” : “proto”` is also
a valid execution requirement, though it’s not required for proto workers,
since they are the default.
You can also set a "worker-key-mnemonic" in the execution requirements. This
may be useful if you're reusing the executable for multiple action types and
want to distinguish actions by this worker.
* Temporary files generated in the course of the action should be saved to the
worker's directory. This enables sandboxing.
Assuming a rule definition with "worker" attribute described above, in addition
to a "srcs" attribute representing the inputs, an "output" attribute
representing the outputs, and an "args" attribute representing the worker
startup args, the call to `ctx.actions.run` might be:
```python
ctx.actions.run(
inputs=ctx.files.srcs,
outputs=[ctx.attr.output],
executable=ctx.attr.worker,
mnemonic="someMnemonic",
execution_requirements={
“supports-workers” : “1”,
“requires-worker-protocol” : “json},
arguments=ctx.attr.args + [“@flagfile”]
)
```
## Examples
The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java),
in addition to an [example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) that is used in our integration tests.
You can use their [scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) to make any Java-based tool into a worker by passing in the correct
callback.
For an example of a rule that uses a worker, take a look at Bazel's
[worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106).
External contributors have implemented workers in a variety of languages; you
can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)!
|
