MUNGE Installation Guide

1. General recommendations

   A. Create a dedicated non-privileged user account

      The munged daemon should be run as a dedicated non-privileged user,
      and its files and directories should be owned by this unique user.
      The recommended user/group name for this account is "munge".  This
      should be a system account with a login shell like "/usr/sbin/nologin"
      to prevent logins.

   B. Maintain consistent UID/GID mappings for users across nodes

      Since authentication is based on UID and GID, users authenticating
      with MUNGE need to have consistent UID/GID mappings across all nodes
      within the security realm.  Since the "munge" user is a system account,
      its UID and GID do not need to be kept in sync across nodes.

   C. Keep system clocks in sync

      MUNGE credentials are valid for a limited time defined by their embedded
      time-to-live value (5 minutes by default).  The fastest and slowest
      system clocks across all nodes within the security realm should be
      within this time interval.

2. Software dependencies

   A. Libgcrypt or OpenSSL

      Either the Libgcrypt or OpenSSL cryptographic library is required.
      Libgcrypt is distributed under LGPLv2.1+.  For the 3.0.0 release,
      OpenSSL switched to the ASLv2 license which is compatible with the
      GPLv3+ license used by MUNGE, but all prior releases are covered by
      the dual OpenSSL and SSLeay license which, on some distributions,
      is incompatible with the GPL.

   B. bzip2

      Support for bzip2 compression will be included if the library is found
      when the software is built.

   C. zlib

      Support for zlib compression will be included if the library is found
      when the software is built.

   D. pkgconf or pkg-config

      A .pc file will be installed if a suitable directory is found or
      specified when the software is built.

3. Building the latest release

   A. Installing from the release tarball

      The release tarball can be verified by its GPG signature or SHA-512
      checksum.  See <https://github.com/dun/munge/wiki/Verifying-Releases>.

      The typical commands "./configure; make; make install" should configure,
      build, and install the software.  Adjust as needed for your environment.

         $ tar xJf munge-0.5.15.tar.xz
         $ cd munge-0.5.15
         $ ./configure \
            --prefix=/usr \
            --sysconfdir=/etc \
            --localstatedir=/var \
            --runstatedir=/run
         $ make
         $ make check
         $ sudo make install

      Note that "configure" will not exist if you have instead downloaded
      a GitHub-generated source code asset (either ".zip" or ".tar.gz").
      The configure script is generated by running "./bootstrap" which
      requires autoconf, automake, and libtool to be installed.

      The configure script has heuristics that attempt to guess the best
      settings for the given system, but the following options allow further
      customization of the installation:

         --with-crypto-lib=(libgcrypt|openssl)
           cryptographic library selection

         --with-logrotateddir=DIR / --without-logrotateddir
           installation directory for logrotate config files

         --with-munge-socket=PATH
           socket pathname default for client/server communication

         --with-pkgconfigdir=DIR / --without-pkgconfigdir
           installation directory for pkg-config .pc files

         --with-runstatedir=DIR
           installation director for modifiable per-process data;
           overrides --runstatedir if both are specified

         --with-sysconfigdir=DIR / --without-sysconfigdir
           installation directory for systemd/sysvinit config files

         --with-systemdunitdir=DIR / --without-systemdunitdir
           installation directory for systemd service unit files

         --with-sysvinitddir=DIR / --without-sysvinitddir
           installation directory for SysV-style init scripts

      The configure "--runstatedir" option appears in autoconf-2.70, and
      was backported to Debian's autoconf-2.69-9.  The "--with-runstatedir"
      option can also be used to specify this directory in case the configure
      script was built by an earlier version of autoconf.

      The "make check" command is optional; it runs the test suite.
      See <https://github.com/dun/munge/wiki/Test-Suite>.

   B. Installing from git

         $ git clone https://github.com/dun/munge.git
         $ cd munge
         $ ./bootstrap
         $ ./configure \
            --prefix=/usr \
            --sysconfdir=/etc \
            --localstatedir=/var \
            --runstatedir=/run
         $ make
         $ make check
         $ sudo make install

      While the release tarball contains "configure", a git checkout does not
      since the autotools-derived products are not under version control.
      The configure script is generated by running "./bootstrap" which
      requires autoconf, automake, and libtool to be installed.

   C. Installing from RPMs on Red Hat compatible systems

      RPMs for recent AlmaLinux, CentOS, and Fedora can be built directly from
      the dist tarball.  Build dependencies can be installed from an SRPM
      if necessary, and the SRPM can be built from the tarball:

         $ rpmbuild -ts munge-0.5.15.tar.xz
         Wrote: SRPMS/munge-0.5.15-1.el9.src.rpm

         $ sudo dnf builddep SRPMS/munge-0.5.15-1.el9.src.rpm
         Package bzip2-devel-1.0.8-8.el9.x86_64 is already installed.
         Package gcc-11.2.1-9.4.el9.alma.x86_64 is already installed.
         Package gnupg2-2.3.3-1.el9.x86_64 is already installed.
         Package make-1:4.3-7.el9.x86_64 is already installed.
         Package openssl-devel-1:3.0.1-23.el9_0.x86_64 is already installed.
         Package procps-ng-3.3.17-4.el9.x86_64 is already installed.
         Package systemd-250-6.el9_0.x86_64 is already installed.
         Package zlib-devel-1.2.11-31.el9_0.1.x86_64 is already installed.

      As of 0.5.14, GPG verification of the source can be enabled by
      specifying "--with=verify" to rpmbuild.  This requires the public key
      <https://github.com/dun.gpg> and corresponding detached GPG signature
      (munge-0.5.15.tar.xz.asc) to reside in the same directory as the
      release tarball (munge-0.5.15.tar.xz):

         $ ls
         dun.gpg  munge-0.5.15.tar.xz  munge-0.5.15.tar.xz.asc

         $ rpmbuild -tb --with=verify munge-0.5.15.tar.xz

      The test suite can be run by specifying "--with=check":

         $ rpmbuild -tb --with=verify --with=check munge-0.5.15.tar.xz

      Or it can be disabled by specifying "--without=check":

         $ rpmbuild -tb --with=verify --without=check munge-0.5.15.tar.xz

      Three or more binary RPMs will be generated: munge, munge-devel,
      munge-libs, and potentially a debugsource and a couple debuginfo RPMs.
      The munge RPM contains the munged daemon, mungekey executable, and
      client executables (munge, unmunge, and remunge).  The munge-devel
      RPM contains the <munge.h> header file for developing applications
      using MUNGE.  The munge-libs RPM contains a shared library for running
      applications that use MUNGE.

      The binary RPMs can be installed with rpm.  For example:

         $ sudo rpm --install --verbose \
            RPMS/x86_64/munge-0.5.15-1.el9.x86_64.rpm \
            RPMS/x86_64/munge-debuginfo-0.5.15-1.el9.x86_64.rpm \
            RPMS/x86_64/munge-debugsource-0.5.15-1.el9.x86_64.rpm \
            RPMS/x86_64/munge-devel-0.5.15-1.el9.x86_64.rpm \
            RPMS/x86_64/munge-libs-0.5.15-1.el9.x86_64.rpm \
            RPMS/x86_64/munge-libs-debuginfo-0.5.15-1.el9.x86_64.rpm

4. Securing the installation

   The munged daemon does not generally require root privileges; see
   <https://github.com/dun/munge/wiki/Auth-Methods>.  If possible, munged
   should be run as a dedicated non-privileged user in accordance with the
   principle of least privilege.

   The munged daemon uses the following system directories (note that
   directories of the form ${somedir} refer to the configure script's
   installation directories and must be substituted accordingly).  Typical
   values when configuring with "--prefix=/usr", "--sysconfdir=/etc",
   "--localstatedir=/var", and "--runstatedir=/run" are shown in brackets.

   A. ${sysconfdir}/munge [/etc/munge]
      This directory will contain the daemon's key.  Its permissions should
      be set to 0700.

   B. ${localstatedir}/lib/munge [/var/lib/munge]
      This directory will contain the daemon's PRNG seed file.  On systems
      where a file-descriptor-passing authentication method is used, this
      is also where the daemon creates pipes for authenticating clients.
      Its permissions should be set to 0711 if using file-descriptor-passing,
      or 0700 otherwise.

   C. ${localstatedir}/log/munge [/var/log/munge]
      This directory will contain the daemon's log file.  Its permissions
      should be set to 0700.

   D. ${runstatedir}/munge [/run/munge]
      This directory will contain the Unix domain socket for clients to
      communicate with the local daemon.  It will also contain the daemon's
      pid file.  This directory must allow execute permissions for all.
      Its permissions should be set to 0755.

   These directories must be owned by the same user as the running daemon
   process.  They cannot allow write permissions for group unless the sticky
   bit is set or the directory is owned by the trusted group (see the munged(8)
   manpage for details on the "--trusted-group" option), and they cannot allow
   write permissions for other unless the sticky bit is set.  In addition,
   all of their parent directories in the path up to the root directory
   must be owned by either root or the same user as the daemon process.
   They cannot allow write permissions for group unless the sticky bit is
   set or the directory is owned by the trusted group, and they cannot allow
   write permissions for other unless the sticky bit is set.

5. Configuration and setup

   A. Creating a key

      All munged daemons within a security realm share a common key.  This key
      is used to cryptographically protect the credential.  Consequently,
      credentials are only valid within a given security realm.

      The "mungekey" executable is the key management utility.  To ensure the
      key file maintains the correct ownership and permissions, it should
      be run by the same user ID that will run the munged daemon process.
      For example, to create a key:

         $ sudo -u munge ${sbindir}/mungekey --verbose

      The key resides in "${sysconfdir}/munge/munge.key".  This file must
      be owned by the same user ID that will run the munged daemon process,
      and its permissions should be set to 0600.  Additionally, this key
      file will need to be securely propagated (e.g., via ssh) to all hosts
      within the security realm.

   B. Setting command-line options

      When starting the daemon via systemd or the init script, command-line
      options to munged can be specified in the OPTIONS line of the
      sysconfig file (typically found in "${sysconfdir}/default/munge" or
      "${sysconfdir}/sysconfig/munge").

6. Starting and stopping the daemon

   The key file "${sysconfdir}/munge/munge.key" must be created before
   starting the daemon.

   A. systemd

      Start the daemon automatically at boot:

         $ sudo systemctl enable munge.service

      Start the daemon now:

         $ sudo systemctl start munge.service

      Stop the daemon:

         $ sudo systemctl stop munge.service

   B. Init script

      Systems utilizing init scripts typically start the daemon by passing
      the "start" command to the script.  The location of the script varies.
      For example:

         $ sudo ${sysconfdir}/init.d/munge start

      Stopping the daemon is done similarly:

         $ sudo ${sysconfdir}/init.d/munge stop

   C. Command-line

      Start the daemon from the command-line so it runs as a non-privileged
      user (e.g., "munge"):

         $ sudo -u munge ${sbindir}/munged

      Stop the daemon with the "--stop" command-line option:

         $ sudo -u munge ${sbindir}/munged --stop

      Or stop the daemon by sending a SIGTERM to the munged process:

         $ sudo -u munge kill $(cat ${runstatedir}/munge/munged.pid)

      Do not stop the daemon by sending a SIGKILL (i.e., kill -9).
      That prevents the daemon from cleaning up -- updating its seed file,
      removing its pid file, removing its socket, etc.

7. Troubleshooting

   A. Verify functionality

      The following steps can be performed to verify that the software is
      properly installed and functioning:

      1. Encode a credential.  This tests if the munge executable and
         libmunge library can be found, if munged is running, and if the
         client (munge/libmunge) can communicate with the server (munged).

            $ munge -n

      2. Encode and decode a credential.  This is similar to the previous test,
         but also tests that the credential has been properly encoded and
         successfully decoded.  Additionally, it shows the metadata that
         has been encoded into the credential.

            $ munge -n | unmunge

      3. Remotely decode a locally-encoded credential.  This tests if
         local and remote munged daemons are running with the same key, if
         the two versions are compatible, if the local defaults for encoding
         the credential can be decoded by the remote daemon, and if the
         clocks between both hosts are within the time interval specified
         in seconds by the -t/--ttl option.

            $ munge -n -t 10 | ssh somehost unmunge

      4. Locally decode a remotely-encoded credential.  This tests if
         local and remote munged daemons are running with the same key,
         if the two versions are compatible, if the remote defaults for
         encoding the credential can be decoded by the local daemon, and if
         the clocks between both hosts are within the time interval specified
         in seconds by the -t/--ttl option.

            $ ssh somehost munge -n -t 10 | unmunge

   B. Check the default locations

      The default locations for the socket, key file, log file, pid file,
      and seed file are configured at build time.  These defaults are shown
      in brackets in the munged "--help" output:

         $ ${sbindir}/munged --help

   C. Check the log

      The munged daemon logs descriptive error messages when possible.
      If munged fails to start, check the log for details.

      1. For systemd, check runtime status information for the munge unit:

         $ sudo systemctl status --full munge.service

      2. For systemd, check the systemd journal:

         $ sudo journalctl -xe | grep munged

      3. For systemd, limit journal output to services run by the munge user:

         $ sudo journalctl _UID=$(id -u munge)

      4. The munged daemon writes to "${localstatedir}/log/munge/munged.log"
         by default; but, the location of this file can be changed with the
         munged "--log-file" option.

      5. If munged is started with the "--syslog" option, log messages are
         instead written to syslog using the "daemon" facility value.
         The name of the corresponding log file will vary depending on the
         syslog configuration.

   D. Run the daemon in the foreground

      If munged fails to start, try running it in the foreground.  When run
      in this manner, log messages are written to stderr.  But remember to
      start munged as the appropriate user:

         $ sudo -u munge ${sbindir}/munged --foreground

   E. "Force" the daemon to run (but use with caution!)

      Some error conditions can be overridden by "forcing" the daemon.
      Use the munged "--force" option to override errors for an existing
      socket, a lack of PRNG entropy, and insecure file/directory permissions.
      But use with caution as overriding these errors can affect security:

         $ sudo -u munge ${sbindir}/munged --force

   F. Common errors

      1. munge: Error: Failed to access "/run/munge/munge.socket.2":
         No such file or directory (Did you start munged?)

         The client was unable to connect to the munged daemon listening
         on the socket "/run/munge/munge.socket.2".  The daemon is likely
         not running.  Try starting it.  If it fails to start, check the
         log for an error message.

      2. munge: Error: Failed to access "/run/munge/munge.socket.2":
         Permission denied

         The client received a "permission denied" error when trying
         to connect to the munged daemon listening on the socket
         "/run/munge/munge.socket.2".  The permissions on the "/run/munge"
         directory are likely incorrect; they should be 0755.  And the socket
         should have permissions 0777.

      3. unmunge: Error: Invalid credential

         The munged daemon decoding the credential is likely using a different
         key than the daemon that encoded it.  First check the key files for
         both daemons.  If they match, try restarting both daemons; since
         the key file is read when the daemon starts, a running daemon could
         be using a key that differs from the current contents of its key file.

      4. unmunge: Error: Expired credential

         The current time (according to the local clock on the host decoding
         the credential) exceeds the creation time of the credential (according
         to the local clock on the host that encoded it) plus its embedded
         time-to-live value.  Either the clocks are out of sync, or too much
         time has passed since the credential was created.

      5. unmunge: Error: Rewound credential

         The current time (according to the local clock on the host decoding
         the credential) precedes the creation time of the credential
         (according to the local clock on the host that encoded it).
         Either the clocks are out of sync, or you've opened a rift in the
         space-time continuum.

      6. unmunge: Error: Replayed credential

         The credential has previously been decoded by this munged daemon.

      7. unmunge: Error: Unauthorized credential for client UID=1234 GID=1234

         Either the UID of the client decoding the credential does not
         match the UID restriction with which the credential was encoded,
         or the GID of the client decoding the credential (or one of its
         supplementary group GIDs) does not match the GID restriction with
         which the credential was encoded.

      8. munged: Error: Failed to find keyfile "/etc/munge/munge.key":
         No such file or directory (Did you run mungekey?)

         A key has not been created.  See mungekey(8).  Note that this
         file will need to be securely propagated to all hosts within the
         security realm.

      9. munged: Error: Found pid 1234 bound to socket
         "/run/munge/munge.socket.2"

         A munged daemon (pid 1234) is already listening on the socket
         "/run/munge/munge.socket.2".  The munged daemon creates the socket
         when it starts, and removes it when it terminates.  While multiple
         munged daemons can run concurrently on the same host, each daemon
         must use a different socket.

8. Using MUNGE

   Applications written in C/C++ can use the interface defined in <munge.h>.
   Compiler and linker flags can be obtained from pkg-config:

      $ cc $(pkg-config --cflags --libs munge) -o foo foo.c

   Scripts can invoke the "munge" and "unmunge" executables -- specify
   "--help" for usage information, or Read The Fantastic Manpages.