1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
|
summary: The shape of the mount namespace on classic systems for non-classic snaps
details: |
This test measures the mount table of the host, of the mount namespace for
a simple core16-based snap for the per-user mount namespace of a simple
core16-base snap as well as for a simple core18 based snap and finally of a
simple snap using classic confinement.
The mount tables are obtained from /proc/pid/mountinfo interface. They are
then fed to mountinfo.query with various determinism adjustment options and
compared to stock tables created when the test was first made.
Naturally occurring mount tables will differ from invocation to invocation
for several reasons. Snap revisions seen in various paths will drift over
time. The order of mount operations that can be done in parallel will
differ from one boot to another. Good examples of that include
/snap/name/revision, which can be all started in parallel. Another one is
the set of control groups mounted in sysfs. Some mount options may be
created based on the amount of available memory. Some mount entries may
contain numeric IDs that are allocated and are hard to predict. Block
devices backing actual file systems may be on various disks, for instance
on /dev/vda or /dev/sdb. All of those are handled by mountinfo.query
--rename and --renumber, along with sorting options that combat
non-deterministic mount order.
Unfortunately this test need separate data sets for google compute engine
and for qemu. The images are just slightly different, containing small
tweaks that result in different initial host mount table. Such changes are
further reflected in per-snap and per-user mount namespaces, since they
contain the view of the host namespace.
Individual backend / system hierarchies can be compared using tools like
meld or even diff. This test is very broad but which makes it somewhat
fragile. It is likely to fail on any change affecting mounts performed by
to snap-confine and snap-update-ns. This is by design. Mount propagation is
somewhat complex and precise tests may not capture broad behavior changes.
Of entire systems.
Lastly, on core systems the test is somewhat artificial as it also measures
changes performed by the test preparation process. Some of the mount
entries are not present on real core systems.
If you see this test randomly failing it may be because it has observed
state leaked by another test that ran on the same machine earlier in the
spread execution chain.
# The test is sensitive to backend type, which designates the used image.
# Backends are enabled one-by-one along with the matching data set.
backends: [google]
# Temporary, exclude on ubuntu-16.04-64 until mount ns changes on 16.04 are understood/fixed
systems: [ubuntu-18.04-64]
# The test itself works perfectly fine but in conjunction with our leaky test
# suite it often fails because it detects cruft left over by other tests in a
# way that was not detected before. Classic systems should be clear of mount
# side-effects now. The test should be _eventually_ enabled on
# ubuntu-core-16-64 and ubuntu-core-18-64.
# We have seen this test to fail when executed after another test that uses LXD
# (eg. tests/main/snap-snap). It is also documented in
# https://bugs.launchpad.net/snapd/+bug/1949710 that LXD modifies the mounts in
# the host system thus making the state of the system different from what is
# expected and checked for in this test. Since we cannot really skip the test,
# cannot set a constraint that the test should not run on the same node as some
# other test, and we cannot fix LXD test to reboot in restore due to
# https://github.com/snapcore/spread/pull/85, we employ a workaround which is to
# bump the priority of this test such that is runs before any other tests that
# may use LXD.
priority: 1001
environment:
MACHINE_STATE/inherit: inherit
MACHINE_STATE/reboot: reboot
prepare: |
# The package qemu-utils creates a tmpfs on /run/qemu on Ubuntu 18.04
# (though not always). This causes failures in this test, as the
# mount-point list is not consistent with what we expect. Therefore,
# unmount it.
umount /run/qemu || true
case "$MACHINE_STATE" in
inherit)
# The test will run with whatever the machine state was originally.
true
;;
reboot)
# TODO: when https://github.com/snapcore/spread/pull/85 is merged
# and released this test can be allowed to run on bash 4.3. Without
# the workaround for a bug in bash REBOOT causes the spread test to
# fail instead of asking spread to reboot the machine.
if "$TESTSTOOLS"/version-compare --strict "$(echo "$BASH_VERSION" | cut -d. -f 1-2)" -eq 4.3; then
echo "SKIP: this test cannot operate on bash 4.3.x"
touch please-skip-this-test
exit 0
fi
#
# The test will reboot once before performing the test. This will
# remove any ephemeral state that may be left in the kernel by prior
# test cases or by project-wide prepare that is does not persist across
# boots.
if [ "$SPREAD_REBOOT" -eq 0 ]; then
REBOOT
fi
;;
esac
# Systemd creates an automount unit for the /proc/sys/fs/binfmt_misc filesystem.
# Any non-special interaction with that directory will trigger auto-mount
# behavior. Since this is extremely easy to do, we don't want to affect
# our measurement by it in any way. A simple way to avoid that is to stop
# the *mount* unit, it will still be auto-mounted on demand but will no
# longer be mounted while we measure.
systemctl stop proc-sys-fs-binfmt_misc.mount
# The --renumber and --rename options renumber and rename various
# non-deterministic elements of the mount table. The --ref-x1000 option
# sets a multiple of 1000 as the base value for allocated renumbered
# identifiers, depending on the depth of "nesting" (via --ref) that is
# used. This makes it easier to maintain the tables in face of changes to
# the host.
#
# The rewrite and display ordering helps with concurrently mounted
# file-systems. This way even if each element inside, say, /snap/... is
# mounted concurrently and may be mounted in different order the measured
# order is deterministic. The use of filesystem field is there so that
# autofs mounted filesystems (like binfmt_misc) don't have random ordering
# between the real thing and the automount entry.
deterministic-mountinfo-query() {
mountinfo.query \
--renumber \
--rename \
--ref-x1000 \
--rewrite-order mount_point \
--rewrite-order mount_source \
--rewrite-order fs_type \
--display-order mount_point \
--display-order mount_source \
--display-order fs_type \
--differential \
.dev .root_dir .mount_point .mount_opts .opt_fields .fs_type .mount_source .sb_opts \
"$@"
}
echo "Install and connect all the test snaps"
# This way the renumbered peer group numbers won't suggest that core18 is
# somehow mounted only in the per-snap mount namespace and the mount
# namespaces of each variant will be more alike since there won't be small
# differences related to set of base snaps installed.
snap pack test-snapd-mountinfo-classic
snap pack test-snapd-mountinfo-core16
snap pack test-snapd-mountinfo-core18
if snap debug sandbox-features --required confinement-options:classic; then
snap install --dangerous --classic test-snapd-mountinfo-classic_1_all.snap
fi
snap install --dangerous test-snapd-mountinfo-core16_1_all.snap
snap install --dangerous test-snapd-mountinfo-core18_1_all.snap
snap connect test-snapd-mountinfo-core16:mount-observe
snap connect test-snapd-mountinfo-core18:mount-observe
echo "Collect mountinfo from the host before running apps"
# "make sure that persistent mount namespaces don't clobber the output"
if mountinfo.query /run/snapd/ns; then
umount /run/snapd/ns
fi
cat /proc/self/mountinfo >HOST.raw.txt
if snap debug sandbox-features --required confinement-options:classic; then
echo "Collect mountinfo from classic, per-snap and per-snap, per-user mount namespaces"
su root -c "snap run test-snapd-mountinfo-classic" >PER-SNAP-C7.raw.txt
su test -c "snap run test-snapd-mountinfo-classic" >PER-USER-C7.raw.txt
fi
echo "Collect mountinfo from core16-based, per-snap and per-snap, per-user mount namespaces"
su root -c "snap run test-snapd-mountinfo-core16" >PER-SNAP-16.raw.txt
su test -c "snap run test-snapd-mountinfo-core16" >PER-USER-16.raw.txt
echo "Collect mountinfo from core18-based, per-snap and per-snap, per-user mount namespaces"
su root -c "snap run test-snapd-mountinfo-core18" >PER-SNAP-18.raw.txt
su test -c "snap run test-snapd-mountinfo-core18" >PER-USER-18.raw.txt
echo "Collect mountinfo from the host after running apps"
# "make sure that persistent mount namespaces don't clobber the output"
snapd.tool exec snap-discard-ns test-snapd-mountinfo-classic
snapd.tool exec snap-discard-ns test-snapd-mountinfo-core16
snapd.tool exec snap-discard-ns test-snapd-mountinfo-core18
umount /run/snapd/ns
cat /proc/self/mountinfo >HOST-AFTER.raw.txt
echo "Transform mountinfo tables to make them deterministic"
deterministic-mountinfo-query -f HOST.raw.txt >HOST.deterministic.txt
deterministic-mountinfo-query --ref HOST.raw.txt -f PER-SNAP-16.raw.txt >PER-SNAP-16.deterministic.txt
deterministic-mountinfo-query --ref HOST.raw.txt --ref PER-SNAP-16.raw.txt -f PER-USER-16.raw.txt >PER-USER-16.deterministic.txt
deterministic-mountinfo-query --ref HOST.raw.txt -f PER-SNAP-18.raw.txt >PER-SNAP-18.deterministic.txt
deterministic-mountinfo-query --ref HOST.raw.txt --ref PER-SNAP-18.raw.txt -f PER-USER-18.raw.txt >PER-USER-18.deterministic.txt
if snap debug sandbox-features --required confinement-options:classic; then
deterministic-mountinfo-query --ref HOST.raw.txt -f PER-SNAP-C7.raw.txt >PER-SNAP-C7.deterministic.txt
deterministic-mountinfo-query --ref HOST.raw.txt --ref PER-SNAP-C7.raw.txt -f PER-USER-C7.raw.txt >PER-USER-C7.deterministic.txt
fi
deterministic-mountinfo-query -f HOST-AFTER.raw.txt >HOST-AFTER.deterministic.txt
if snap debug sandbox-features --required confinement-options:classic; then
snap remove --purge test-snapd-mountinfo-classic
fi
snap remove --purge test-snapd-mountinfo-core16
snap remove --purge test-snapd-mountinfo-core18
debug: |
for fname in ./*.deterministic.txt; do
echo
cat "$fname"
echo
done
execute: |
if [ -e please-skip-this-test ]; then
exit 0
fi
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/HOST.expected.txt" HOST.deterministic.txt
# The before and after host files should be identical.
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/HOST.expected.txt" HOST-AFTER.deterministic.txt
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-SNAP-16.expected.txt" PER-SNAP-16.deterministic.txt
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-USER-16.expected.txt" PER-USER-16.deterministic.txt
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-SNAP-18.expected.txt" PER-SNAP-18.deterministic.txt
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-USER-18.expected.txt" PER-USER-18.deterministic.txt
if snap debug sandbox-features --required confinement-options:classic; then
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-SNAP-C7.expected.txt" PER-SNAP-C7.deterministic.txt
diff -u "$SPREAD_BACKEND.$SPREAD_SYSTEM/PER-USER-C7.expected.txt" PER-USER-C7.deterministic.txt
fi
|
