Package: ikiwiki-hosting / 0.20180719-2

Metadata

Package Version Patches format
ikiwiki-hosting 0.20180719-2 3.0 (quilt)

Patch series

view the series file
Patch File delta Description
Avoid directly running init scripts instead use the servi.patch | (download)

CHANGELOG | 6 6 + 0 - 0 !
ikidns | 2 1 + 1 - 0 !
ikiwiki-hosting-web-daily | 2 1 + 1 - 0 !
3 files changed, 8 insertions(+), 2 deletions(-)

 avoid directly running init scripts,
 instead use the service command.

For one thing, service is standard and may be more portable.

But the real reason is I have seen apache2 processes somehow end up not
running in the apache2 cgroup, and so it seems that somehow something that
is restarting it doesn't end up using systemctl. That breaks eg, the
preiodic reloads of apache2 to see new letsencrypt certs.

/etc/init.d/apache uses the lsb init-functions, which normally when
loaded realize systemd is in use and bypass the usual init script code.

HOWEVER, in /lib/lsb/init-functions.d/40-systemd there is this mess:

    # Redirect SysV init scripts when executed by the user
    if [ $PPID -ne 1 ] && [ -z "${SYSTEMCTL_SKIP_REDIRECT:-}" ]; then
        case $(readlink -f "$executable") in
            /etc/init.d/*)
                # If the state is not-found, this might be a newly installed SysV init
                # script where systemd-sysv-generator has not been run yet.
                [ "$state" != "not-found" ] || [ "$(id -u)" != 0 ] || systemctl --no-ask-password daemon-reload

                _use_systemctl=1
                # Some services can't reload through the .service file,
                # but can through the init script.
                if [ "$(systemctl -p CanReload --value show $service 2>/dev/null)" = "no" ] && [ "${argument:-}" = "reload" ]; then
                    _use_systemctl=0
                fi

If that doesn't set _use_systemctl=1, systemctl will not be used and the
old init script code will be run. It might kill the apache2 processes
owned by systemd and start up the daemon in some other control group
(possibly cron's when a cron job ran the init script).

Seems unlikely that PPID would be 1 for a cron job. It's at least
theoretically posible that the init script might not exist in
/etc/init.d, if apache happens to get upgraded just as a cron job is
running its init script and dpkg doesn't update the conffile
atomically. Or perhaps systemctl -p CanReload is for whatever reason
no at some point in time.

Whatever it is, it must only happen intermittently, because
the ikiwiki-hosting-web-daily cron job contained a
/etc/init.d/apache2 reload
and yet apache was not restarted for months on our server. So at least
most of the time that must have been trying to restart it via systemctl
and failing because it had already escaped its control.

My takeaway is that this is almost certianly why we're getting apache2
processes outside the control of systemd, and the best way to avoid it,
short of tracking down whatever the bug is, is to avoid running init
scripts by hand and use systemctl, or something that runs it.

(Only other possibility might be that some admin did something stupid
like manually killing and starting apache, but I can find no evidence of
that in the logs.)

Hopefully using `service` will avoid the problem. It detects systemd in a
less byzantine way and should always use systemctl for start and stop.
Although, for reload, it does also check systemctl -p CanReload and falls
back to running the init script, so if that is somehow the root cause it
would still be a problem.

(cherry picked from commit 27927bfd630dfb09fed9d27b2c9ed37e90c0eaa3)

Avoid running apache2ctl graceful which may restart apach.patch | (download)

CHANGELOG | 2 2 + 0 - 0 !
ikisite | 35 28 + 7 - 0 !
2 files changed, 30 insertions(+), 7 deletions(-)

 avoid running apache2ctl graceful which may restart apache,
 instead use the service command
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

Following on from 27927bfd630dfb09fed9d27b2c9ed37e90c0eaa3, I reproduced
the problem again and it looked like this:

   CGroup: /
           ├─init.scope
           │ └─1 /lib/systemd/systemd --system --deserialize 25
           └─system.slice
             ├─ssh.service
             │ ├─  640 /usr/sbin/apache2 -k graceful
             │ ├─  816 /usr/sbin/apache2 -k graceful
             │ ├─ 1133 /usr/sbin/apache2 -k graceful
             │ ├─ 1140 /usr/sbin/apache2 -k graceful
             │ ├─ 1298 /usr/sbin/apache2 -k graceful
             │ ├─ 1599 /usr/sbin/apache2 -k graceful

There were dozens of the processes which had all been started in a small amount
of time. I was upgrading ikiwiki-hosting at the time, so it seems likely that
ikisite ran it for each of the dozens of sites. Seems to point the finger at
apache2ctl graceful, but I don't reproduce the problem running it manually.

My guess is that, in some circumstances, apache2ctl graceful fails to
talk to the currently running apache daemon, and so I guess proceeds to
kill it and starts a new one, thus escaping the systemd cgroup.

If so, this change should avoid that, while keeping the graceful detection of
broken apache configs.

(cherry picked from commit 49956083ada2efe59ddf497182e7e1d2717b2a49)

shut up apache2ctl configtest output.patch | (download)

ikisite | 2 1 + 1 - 0 !
1 file changed, 1 insertion(+), 1 deletion(-)

 shut up apache2ctl configtest output

(cherry picked from commit 75032c8fe4b12163a9b1f8470cc876be36a778d1)