1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195
|
summary: Check that refreshing snapd does not interfere with snap services
details: |
This test verifies the fix for the issue raised in https://bugs.launchpad.net/snapd/+bug/1924805
which fixes a race condition on certain devices where the device unexpectedly reboots during a
snapd refresh.
The problem is reproduced when we refresh snapd itself after having written out some systemd
units that include the Requires= above (which could happen for app snap refreshes or new
installs after snapd 2.49.2 is installed), then unmount/stop usr-lib-snapd.mount which,
due to how Requires= works, will forcibly stop all services which have the Requires= bit in it.
Then, after snapd finishes refreshing itself, those services will be left in the
terminated state, effectively killing the services when snapd refreshes itself.
# TODO: we should also run it on classic later
systems:
# snapd 2.49.2 doesn't support snap-declaration format 5, which is used by the
# kernel on core18 systems
# - ubuntu-core-18-*
- ubuntu-core-20-*
# TODO: 2.49.2 does not work on UC22 because of ABI breakage and missing
# libgcc_s.so.1
# - ubuntu-core-22-*
environment:
# uploading the snapd snap triggers OOM
SNAPD_NO_MEMORY_LIMIT: 1
SNAPD_VERSION_UNDER_TEST/start_w_pr: pr
SNAPD_VERSION_UNDER_TEST/start_w_stable: stable
SNAPD_VERSION_UNDER_TEST/start_w_2_49_2: "2.49.2"
# links to specific snapd versions
SNAPD_2_49_1_X86: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.1_11402.snap
SNAPD_2_49_1_ARM64: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.1_11408.snap
SNAPD_2_49_1_ARMHF: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.1_11410.snap
SNAPD_2_49_2_X86: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.2_11588.snap
SNAPD_2_49_2_ARM64: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.2_11584.snap
SNAPD_2_49_2_ARMHF: https://storage.googleapis.com/snapd-spread-tests/snaps/snapd_2.49.2_11586.snap
prepare: |
# save the current version of snapd for later
INITIAL_REV=$(snap list snapd | tail -n +2 | awk '{print $3}')
cp "/var/lib/snapd/snaps/snapd_$INITIAL_REV.snap" snapd-pr.snap
snap set system experimental.parallel-instances=true
tests.cleanup defer snap unset system experimental.parallel-instances
# keep around all the snapd snap revisions we will use in the test so that we
# can always easily revert back to the one at the end of the test
snap set system refresh.retain=5
tests.cleanup defer snap unset system refresh.retain
execute: |
# check if snapd 2.49.2 is the current latest/stable release as it simplifies
# some of the logic below
if snap info snapd | gojq --yaml-input -r '.channels."latest/stable"' | grep -q -Po '2.49.2\s+'; then
# skip the stable variant of the test
if [ "${SNAPD_VERSION_UNDER_TEST}" = "stable" ]; then
echo "Skipping duplicated test case"
exit 0
fi
fi
if ! os.query is-pc-amd64 && ! os.query is-arm; then
echo "architecture not supported for this variant"
exit 0
fi
echo "Ensure that the system is fully seeded"
snap changes | MATCH "Done.*Initialize system state"
INITIAL_REV=$(snap list snapd | tail -n +2 | awk '{print $3}')
# first thing is to install snapd 2.49.1 before the Requires= change was
# introduced so we can install a snap service that will not have Requires= in
# it
if os.query is-pc-amd64; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.1.snap "$SNAPD_2_49_1_X86"
elif os.query is-arm64; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.1.snap "$SNAPD_2_49_1_ARM64"
elif os.query is-armhf; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.1.snap "$SNAPD_2_49_1_ARMHF"
fi
snap install --dangerous snapd_2.49.1.snap
snap version | MATCH 2.49.1
# always go back to the original revision from the pr at the end of the test
tests.cleanup defer snap revert snapd --revision="$INITIAL_REV"
echo "Install a service from snapd 2.49.1 to have one without Requires= in it"
"$TESTSTOOLS"/snaps-state install-local test-snapd-simple-service
# check that it is initially active
snap services|MATCH ".*test-snapd-simple-service\s*enabled\s*active.*"
# check that it doesn't have any dependencies on usr-lib-snapd.mount at the
# start
NOMATCH Requires=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service.test-snapd-simple-service.service
NOMATCH Wants=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service.test-snapd-simple-service.service
# now refresh to the variant of the test
if [ "${SNAPD_VERSION_UNDER_TEST}" = "stable" ]; then
echo "Refreshing snapd to stable"
snap refresh --amend --channel=latest/stable snapd
elif [ "${SNAPD_VERSION_UNDER_TEST}" = "2.49.2" ]; then
# download and install snapd 2.49.2
if os.query is-pc-amd64; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.2.snap "$SNAPD_2_49_2_X86"
elif os.query is-arm64; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.2.snap "$SNAPD_2_49_2_ARM64"
elif os.query is-armhf; then
"$TESTSTOOLS/simpleget" -o snapd_2.49.2.snap "$SNAPD_2_49_2_ARMHF"
fi
echo "Refreshing snapd to 2.49.2"
snap install --dangerous snapd_2.49.2.snap
snap version | MATCH 2.49.2
elif [ "${SNAPD_VERSION_UNDER_TEST}" = "pr" ]; then
# refresh back to the version we originally had from before the test
# started
echo "Refreshing snapd to version from the pr"
snap install --dangerous snapd-pr.snap
fi
# now install another service that will either get Requires= or Wants= for
# usr-lib-snapd.mount, depending on the variant of the test
"$TESTSTOOLS"/snaps-state install-local-as test-snapd-simple-service test-snapd-simple-service_alt
# check that it is still initially active
snap services|MATCH ".*test-snapd-simple-service_alt.test-snapd-simple-service\s*enabled\s*active.*"
# if we are not running with 2.49.2, then the second service should have been
# generated with Wants=, but if we are doing 2.49.2 then the second service
# should have Requires= in it
if [ "${SNAPD_VERSION_UNDER_TEST}" = "2.49.2" ];then
MATCH Requires=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service_alt.test-snapd-simple-service.service
else
MATCH Wants=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service_alt.test-snapd-simple-service.service
fi
# now repack current snapd to refresh to it again and observe the results
unsquashfs -d ./snapd-unpacked snapd-pr.snap
snap pack --filename=snapd_repacked.snap snapd-unpacked
# save the PID's of the services
systemctl show -p MainPID snap.test-snapd-simple-service.test-snapd-simple-service > old-main.pid
systemctl show -p MainPID snap.test-snapd-simple-service_alt.test-snapd-simple-service > old-main_alt.pid
echo "Refresh snapd"
snap install --dangerous snapd_repacked.snap
# for all variants of the test, the main.pid should be the same since it did
# not ever contain the Requires=
systemctl show -p MainPID snap.test-snapd-simple-service.test-snapd-simple-service > new-main.pid
if [ "$(cat new-main.pid)" != "$(cat old-main.pid)" ]; then
echo "The service without Requires= was restarted; test is broken"
exit 1
fi
# for the variants of the test that have the fix in them, that is stable and
# and pr, then we also should not have had the alt service pid change due to a
# restart
# for the other variant (2.49.2 only), we unfortunately ended up needing to
# restart the service so it should have a different PID, but it should be
# running again
systemctl show -p MainPID snap.test-snapd-simple-service_alt.test-snapd-simple-service > new-main_alt.pid
if [ "${SNAPD_VERSION_UNDER_TEST}" = "2.49.2" ]; then
if [ "$(cat new-main_alt.pid)" = "$(cat old-main_alt.pid)" ]; then
echo "Somehow the service was not killed as expected ... test is probably broken"
exit 1
fi
else
if [ "$(cat new-main.pid)" != "$(cat old-main.pid)" ]; then
echo "The service with Wants= was unexpectedly killed; test is broken"
exit 1
fi
fi
# in all cases both services should be active after the refresh
echo "Check services were kept active"
snap services|MATCH ".*test-snapd-simple-service\s*enabled\s*active.*"
snap services|MATCH ".*test-snapd-simple-service_alt.test-snapd-simple-service\s*enabled\s*active.*"
# and both services should have Wants= now
echo "Check services were re-written to use Wants=usr-lib-snapd.mount now"
MATCH Wants=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service_alt.test-snapd-simple-service.service
MATCH Wants=usr-lib-snapd.mount < /etc/systemd/system/snap.test-snapd-simple-service.test-snapd-simple-service.service
|