1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
Multi-repository configuration management: `just-lock`
======================================================
The multi-repository build configuration used as input by `just-mr` acts for
all intents and purposes as a _lockfile_ for *justbuild* projects, containing
the pinned versions of all content-fixed repositories. This is expected to be
stored and shipped together with the source code, allowing a consistent
development environment for all users.
With dependencies of projects coming in many shapes and forms, `just-lock` is
a tool for generating and maintaining the multi-repositories configuration
(lock)file of *justbuild* projects. This tool performs several functionalities,
such as importing dependencies with repository composition (extending the
functionality available in the `just-import-git` tool), automatic deduplication
of identical transitive dependencies (a functionality available also standalone
in the `just-deduplicate-repos` tool), or setup of repository clones for local
development. For the purposes of this tutorial, we will focus only on the
dependency import aspects.
Basic use of `just-lock`
------------------------
In order to produce the multi-repository build configuration file, `just-lock`
expects an input configuration file. The format of this file is an extension of
the one of `just-mr` and it is read as a `JSON` object. The file defines four
elements:
- the description of local repositories, as the value for mandatory field
`"repositories"`. This usually defines local checkouts, patches, overlays.
- the description of remote dependencies, as the optional value for field
`"imports"`. For *justbuild* dependencies, their multi-repository configuration
(lock)file is taken as the ground truth for importing repository descriptions
into the current project. Non-*justbuild* dependencies can also be described
here, but can only be imported as-is, with overlays defining how such source
code should be integrated (as it will be shown below).
- the repository to consider as the main entry point for the build, as the
value of the optional field `"main"`.
- a list of repository names, as the value of the field `"keep"`. These names
will be kept in the final configuration during the automatic process of
deduplicating any repeated repositories brought in as transitive dependencies
from different imports. For the purposes of this tutorial, the use of this
field is not exemplified.
### The input file
For the following, we return to the *hello_world* example from the section on
[*Building Third-party dependencies*](./third-party-software.md), which depends
on the open-source project [fmtlib](https://github.com/fmtlib/fmt), in the
setup that enables high-level target caching.
We define the following `repos.in.json` input configuration file for
`just-lock`:
``` {.jsonc srcname="repos.in.json"}
{ "main": "tutorial"
, "repositories":
{ "rules-cc":
{ "repository": "rules-cc-rules-sources"
, "target_root": "tutorial-defaults"
, "rule_root": "rules-cc"
}
, "tutorial":
{ "repository": {"type": "file", "path": "."}
, "bindings": {"rules": "rules-cc", "format": "fmtlib"}
}
, "tutorial-defaults":
{ "repository":
{ "type": "file"
, "path": "./tutorial-defaults"
, "pragma": {"to_git": true}
}
}
, "fmt-targets-layer":
{ "repository":
{ "type": "file"
, "path": "./fmt-layer"
, "pragma": {"to_git": true}
}
}
, "fmtlib":
{ "repository": "fmtlib-sources"
, "target_root": "fmt-targets-layer"
, "bindings": {"rules": "rules-cc"}
}
}
, "imports":
[ { "source": "git"
, "branch": "master"
, "commit": "7a2fb9f639a61cf7b7d7e45c7c4cea845e7528c6"
, "url": "https://github.com/just-buildsystem/rules-cc.git"
, "repos": [{"alias": "rules-cc-rules-sources", "repo": "rules"}]
}
, { "source": "git"
, "branch": "8.1.1"
, "url": "https://github.com/fmtlib/fmt.git"
, "repos": [{"alias": "fmtlib-sources"}]
, "as plain": true
}
]
}
```
As already mentioned, the `"imports"` field provides a description of the
external (usually third-party) dependencies of a project. In our case, we
use the `rules-cc` and `fmtlib` libraries, described as `git` sources.
The first thing to note is the use of overlays. In our simple project, as we
do not want to import anything other than the source trees of the needed
dependencies, we create the `"rules-cc-rules-sources"` and `"fmtlib-sources"`
names, which will bind the workspace roots of our `"rules-cc"` and `"fmtlib"`
repositories, respectively, to the descriptions of the remote repositories
providing the necessary source trees.
The first import description object is for `rules-cc`, which is a *justbuild*
project hosted as a Git repository, from which we would like to import its
`"rules"` subdirectory. Thankfully, that project offers a useful shorthand by
defining in its own locked repositories configuration file the `"rules"` overlay
repository pointing to the respective subdirectory. `just-lock` will read that
configuration file in order to produce the resulting configuration, caching any
fetched source trees. It is thus highly recommended that the same build root as
the one subsequently used by `just-mr` is provided to `just-lock` (via the
`--local-build-root` option, with same default behaviour as in `just-mr`).
In the case of `fmtib`, which is also hosted as a Git repository but does not
provide a *justbuild* configuration file, we can only import it as-is, as a
complete repository, signaled by setting the `"as plain"` flag. Do note that in
this case we can limit ourselves to also just providing the `"branch"` field,
and not also a specific commit. This is because `just-lock` automatically
interrogates the remote in order to retrieve the top commit associated to a
certain reference (in this case, a release tag) and pin it to a hard reference
(in this case, the commit identifier) into the output configuration. This is a
useful feature, as it is often the case that information about dependencies
comes in the form of "loose" references, such as release tags or even simply the
remote location of a distfile, but which `just-lock` then will pin down to a
content-defined reference, such as a commit, blob, or tree identifier.
### Generating the configuration
We generate an output configuration file `repos.out.json` by running `just-lock`
with the appropriate arguments, then we build the `helloworld` target with this
configuration:
``` sh
$ just-lock -C repos.in.json -o repos.out.json
[...]
$
$ just-mr -C repos.out.json build helloworld
INFO: Performing repositories setup
INFO: Found 5 repositories involved
INFO: Setup finished, exec ["just","build","-C","...","helloworld"]
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 4 actions, 0 tree overlays, 2 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 4 actions, 4 cache hits.
INFO: Artifacts built, logical paths are:
helloworld [18d25e828a0176cef6fb029bfd83e1862712ec87:132736:x]
$
```
As it can be seen, everything comes from cache and the new configuration behaves
identical to the manually written one. If available, one can inspect the
difference in content between the two files using the `jq` command-line tool:
``` sh
$ diff -y <(jq --sort-keys . repos.out.json) <(jq --sort-keys . repos.json)
{ {
"main": "tutorial", "main": "tutorial",
"repositories": { "repositories": {
"fmt-targets-layer": { "fmt-targets-layer": {
"repository": { "repository": {
"path": "./fmt-layer", "path": "./fmt-layer",
"pragma": { "pragma": {
"to_git": true "to_git": true
}, },
"type": "file" "type": "file"
} }
}, },
"fmtlib": { "fmtlib": {
"bindings": { "bindings": {
"rules": "rules-cc" "rules": "rules-cc"
}, },
"repository": "fmtlib-sources", <
"target_root": "fmt-targets-layer" <
}, <
"fmtlib-sources": { <
"repository": { "repository": {
"branch": "8.1.1", "branch": "8.1.1",
"commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9", "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9",
"repository": "https://github.com/fmtlib/fmt.git", "repository": "https://github.com/fmtlib/fmt.git",
"type": "git" "type": "git"
} | },
> "target_root": "fmt-targets-layer"
}, },
"rules-cc": { "rules-cc": {
"repository": "rules-cc-rules-sources", <
"rule_root": "rules-cc", <
"target_root": "tutorial-defaults" <
}, <
"rules-cc-rules-sources": { <
"repository": { "repository": {
"branch": "master", "branch": "master",
"commit": "7a2fb9f639a61cf7b7d7e45c7c4cea845e7528c6", "commit": "7a2fb9f639a61cf7b7d7e45c7c4cea845e7528c6",
"repository": "https://github.com/just-buildsystem/ru "repository": "https://github.com/just-buildsystem/ru
"subdir": "rules", "subdir": "rules",
"type": "git" "type": "git"
} | },
> "rule_root": "rules-cc",
> "target_root": "tutorial-defaults"
}, },
"tutorial": { "tutorial": {
"bindings": { "bindings": {
"format": "fmtlib", "format": "fmtlib",
"rules": "rules-cc" "rules": "rules-cc"
}, },
"repository": { "repository": {
"path": ".", "path": ".",
"type": "file" "type": "file"
} }
}, },
"tutorial-defaults": { "tutorial-defaults": {
"repository": { "repository": {
"path": "./tutorial-defaults", "path": "./tutorial-defaults",
"pragma": { "pragma": {
"to_git": true "to_git": true
}, },
"type": "file" "type": "file"
} }
} }
} }
} }
```
Except for the two overlays, kept from the `just-lock` input file configuration,
the two configuration files have the same content.
|
