1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
|
Document by Ximin Luo, Luca Bruno, Sylvestre Ledru & Fabian Grünbichler
This source package is unfortunately quite tricky and with several cutting
edges, due to the complexity of rust-lang bootstrapping system and the high
rate of language changes still ongoing.
We try to describe here inner packaging details and the reasons behind them.
If you are looking to help maintain this package, be sure to read the "Notes
for package maintainers" section further below.
Embedded libraries
==================
The upstream source package embeds many external libraries. We make a great
effort to remove them and use system versions where possible, but there are a
few more remaining:
* vendor/dlmalloc
These are small C libraries designed to be statically linked; their upstream
does not support building them as a shared library and they are too small to
justify their own Debian package.
Building from source
====================
The Debian rustc package will use the system rustc to bootstrap itself from.
The system rustc has to be either the previous or the same version as the rustc
being built; the build will fail if this is not the case.
sudo apt-get build-dep ./
dpkg-buildpackage
# Or, to directly use what's in the Debian FTP archive
sudo apt-get build-dep rustc
apt-get source --compile rustc
Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
instead use the process defined by Rust upstream. This downloads the "official"
stage0 compiler for the version being built from rust-lang.org. At the time of
writing "official" means "the previous stable version".
sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
dpkg-buildpackage -P pkg.rustc.dlstage0
# Or, to directly use what's in the Debian FTP archive
sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
apt-get source --compile -P pkg.rustc.dlstage0 rustc
After [1] is fixed, both of these should in theory give identical results.
If neither of these options are acceptable to you, e.g. because your distro
does not have rustc already and your build process cannot access the network,
see "Bootstrapping" below.
[1] https://github.com/rust-lang/rust/issues/34902
Bootstrapping
=============
To bootstrap rustc on a distro that does not have it or cargo available on any
architecture (so cross-compiling is not an option) you can run `debian/rules
source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
cargo. Instead, it includes an extra orig-stage0 source tarball that contains
the official stage0 compiler, pre-downloaded from rust-lang.org so that your
build daemons don't need to access the network during the build.
debian/rules source_orig-stage0
# Follow the final manual instructions that it outputs. Then:
sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
To only bootstrap specific architectures, run this instead:
upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
This way, other architectures will be omitted from the orig-stage0 tarball. You
might want to do this e.g. if these other architectures are already present in
your distro, but the $upstream_bootstrap_arch ones are not yet present.
If the toolchain for the architecture you are attempting to bootstrap is not
provided upstream (i.e., it's not at Tier 2 with Host Tools or higher[2]), you
can manually prepare such a stage0 tarball via cross compilation using
upstream's build process.
[2] https://doc.rust-lang.org/nightly/rustc/platform-support.html
Notes
-----
The approach bundles the upstream bootstrapping binaries inside the Debian
source package. This is a nasty hack that stretches the definition of "source
package", but has a few advantages explained below.
The traditional Debian way of bootstrapping compilers - and other distros have
similar approaches - is some variant of the following:
1. A developer locally installs some upstream bootstrapping binaries.
2. They locally build a Debian package, using these binaries as undeclared
build dependencies.
3. They upload these binary packages to Debian, which can be used as declared
Build-Depends in the future, including by the same package.
The problem with this is, Debian does not have any policy nor infrastructure
that can try to reproduce what this developer supposedly did.
Using bootstrapping binary blobs *at some point of the process* is unavoidable.
Rather than pretending we didn't do this, it is better to record *which blobs*
we used, so it can be audited later. If we bundle non-Debian build-dependencies
inside the source package, then we can do a *source-only upload*, and the
building of the binary packages can be done by the normal build infrastructure.
If the build process is reproducible [1] then we can be sure that *you* (as the
developer that prepared the source-only upload) didn't backdoor the binaries,
nor did the build daemons even if they were compromised during the build.
The bootstrapping binaries may still have been backdoored, but this is true in
both scenarios. So our arrangement is still a strict improvement in security,
because it reduces the set of "things that may have been backdoored". Also,
more people use the upstream binaries than the "magical original Debian
package", so backdoors have a greater chance of being detected in the former.
In the long run, this process is laying the foundations for doing Diverse
Double-Compilation [2], where we use *many independent* bootstrapping binaries
to reproduce bit-for-bit identical output compilers, giving confidence that
nothing was backdoored along the way.
[1] The build process for rustc is currently *not* reproducible but we're
working towards it. https://github.com/rust-lang/rust/issues/34902
[2] http://www.dwheeler.com/trusting-trust/
Maintaining this package
========================
Import of a new upstream version
--------------------------------
$ apt install equivs python3-magic
$ sudo mk-build-deps -irt 'aptitude -R'
$ uscan --verbose # or debian/rules source_orig-beta, for beta
$ ver=UPDATE-ME # whatever it is, probably X.YY.Z or X.YY.Z~beta.N
$ debian/rebase-patches.sh $ver
# This will require an understanding of how git-rebase and git-mergetool works
# We recommend either kdiff3 or p4merge (proprietary) as the git-mergetool.
# See individual patches for instructions on rebasing.
$ tar -C /tmp -xf ../rustc-${ver/\~/-}-src.tar.xz && ( dir=$PWD; cd /tmp/rustc-${ver/*~*/beta}-src/ && pwd && $dir/debian/prune-unused-deps ) && rm -rf /tmp/rustc-${ver/*~*/beta}-src/
$ git diff
# Review the diff. If it removes too much stuff, it could mean that rustc
# pulled in new unnecessary dependencies in this newer version. See if you can
# drop them by amending the patch "d-0000-ignore-removed-submodules.patch".
# Rerun the above "tar ..." commands again and check that your patch works.
# For example, there is absolutely no reason to pull in windows-sys/windows or
# openssl-src.
$ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright
$ uscan --verbose # yes, again, to pick up the new Files-Excluded stuff
# or debian/rules source_orig-beta, for beta
# Keep running this and follow its instructions, until it gives no output:
$ debian/check-orig-suspicious.sh $ver
# When you are satisfied with the above, proceed:
$ git checkout debian/experimental
$ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
$ dch -v $ver+dfsg1-1~exp1 "New upstream release."
$ debian/rules update-version
# then refresh patches, etc etc
# Use /usr/share/cargo/scripts/guess-crate-copyright to help update d/copyright quickly
# If you need to repack again, bump the 'repacksuffix' in d/watch then run
$ uscan --verbose --force-download
# This will do a local repack using the new Files-Excluded rules, without
# redownloading the orig tarball (despite the slightly misleading flag).
Proceeding after build failure
------------------------------
If your build fails, don't run `./x.py` directly as that will detect it's being
run with different settings, and run the build from scratch all over again.
overwriting all intermediate files. Instead, do:
$ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever"
Hopefully, this will directly proceed to the step that failed, without
rebuilding everything in between.
Comparing Debian rustc vs upstream rustc
----------------------------------------
This package does things the Debian way, which differs significantly from
upstream practices. If you find a bug, you might want to check if it is present
in the upstream package. Run "debian/rules debian/config.toml" to generate our
config.toml that you can then use in an upstream directory **unpacked from the
release tarball*. (It is more complex to get this working with their git repo.)
This will configure it in a "halfway" style between upstream and Debian.
Specifically, it will not build LLVM nor download stuff from crates.io, yet
Debian patches are *not* applied. These specific settings were chosen as a
tradeoff between convenience vs being close to what upstream does - so that the
chances of a bug here being a genuine upstream issue rather than a Debian bug,
is much higher. Also, with the exception of LLVM, these are non-default modes
*supported by* upstream so they would be happy to receive bug reports about it
even if your issue only occurs here.
OTOH if you need to test a completely clean upstream build, including all the
annoying stuff like building LLVM and downloading dependencies from crates.io,
simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
This can be useful for confirming that an issue is caused by Debian's LLVM.
If you need to test a LLVM patch, do something like this:
# build your patched LLVM debs, then:
$ mkdir -p llvm-destdir && cd llvm-destdir
$ ver=4.0; VERSION=FIXME
$ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \
dpkg -x ../"$i"_*${VERSION}_*.deb .; done
$ cd ../rustc
$ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build
If you need to test a patch to the stage0 rustc, do something like this:
# build your patched rustc debs or upstream rustc, then:
$ mkdir -p rust-destdir && cd rust-destdir
$ ver=1.20; VERSION=FIXME;
$ for i in rustc libstd-rust-$ver libstd-rust-dev; do \
dpkg -x ../"$i"_*${VERSION}_*.deb .; done
$ cd ../rustc
$ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build
Useful links
------------
The Fedora rust team is more active than the Debian one. Here are their links:
Source code
https://src.fedoraproject.org/rpms/rust/tree/
Binary packages and test logs
https://kojipkgs.fedoraproject.org//packages/rust/
If the same test fails both on Fedora and Debian it's a good indication that
we're not Doing It Wrong and can file a valid bug upstream.
|
