---------------------- Tuptime Manual ---------------------- version 5.2.2 Ricardo F. 06/Jan/2023 ============ | Abstract | ============ Tuptime report historical and statistical real time of the system, keeping it between restarts. Indeed, it can: - Count system startups - Register first boot time (since installation time) - Count nicely and accidentally shutdowns - Uptime and downtime percentage since first boot time - Accumulated system uptime (running and sleeping), downtime and total - Register used kernels and boot IDs - Report current uptime - Print formatted table or list with the system history - Narrow reports since, until or at a given startup or timestamp - Output in csv format ================= | Prerequisites | ================= Operating systems: - Linux - BSD Software: - Python 3.x - Python modules (most included in python core by default): sys, os, argparse, locale, platform, signal, logging, sqlite3, datetime If a distro is used, all is met with: - In Linux: 'python' package >= 3.0 - In FreeBSD: 'python3' and 'py36-sqlite3' ================ | Installation | ================ For Linux systems: + From distribution repository: Debian - https://packages.debian.org/tuptime Ubuntu - https://packages.ubuntu.com/tuptime # apt install tuptime Fedora, EPEL - https://src.fedoraproject.org/rpms/tuptime # dnf install tuptime # systemctl enable tuptime.service && systemctl start tuptime.service # systemctl enable tuptime-sync.timer && systemctl start tuptime-sync.timer + Using bash script: Download and execute the installation script: # bash < <(curl -Ls https://git.io/tuptime-install.sh) + Manual install: Clone repository and copy executable file: # install -m 755 tuptime/src/tuptime /usr/bin/tuptime Add _tuptime user with sysusers or useradd, choose one: With sysusers: # install -m 644 tuptime/src/systemd/tuptime.sysusers /usr/lib/sysusers.d/tuptime.conf # systemd-sysusers /usr/lib/sysusers.d/tuptime.conf With useradd: # useradd --system --no-create-home --home-dir '/var/lib/tuptime' \ --shell '/bin/false' --comment 'Tuptime execution user' _tuptime Execute tuptime with a privilege user for create DB path: # tuptime Change owner of the DB path and file: # chown -R _tuptime:_tuptime /var/lib/tuptime Add sync execution with cron or timer, choose one: With cron file: # install -m 644 tuptime/src/cron.d/tuptime /etc/cron.d/tuptime With systemd timer: # install -m 644 tuptime/src/systemd/tuptime-sync.* /lib/systemd/system/ # systemctl enable tuptime-sync.timer && systemctl start tuptime-sync.timer Copy systemd service file and enable it: # install -m 644 tuptime/src/systemd/tuptime.service /lib/systemd/system/tuptime.service # systemctl enable tuptime.service && systemctl start tuptime.service Or if it is an old RedHat system or derivate, copy the init file: # install -m 755 tuptime/src/init.d/redhat/tuptime /etc/init.d/tuptime # chkconfig --add tuptime && chkconfig tuptime on Or if it is and old Debian system or derivate, copy the init file: # install -m 755 tuptime/src/init.d/debian/tuptime /etc/init.d/tuptime # update-rc.d tuptime defaults # /etc/init.d/tuptime start That's all, enjoy it. For BSDs systems: + On FreeBSD and derivates using packages or ports: # pkg install tuptime (or) # /usr/ports/sysutils/tuptime Add the suggested cron entry and enable it on rc: # sysrc tuptime_enable=YES # service tuptime start + Manual install on FreeBSD, OpenBSD, NetBSD and derivates: Please, read misc/bsd-manual-install.txt Partial support for OSX (Darwin) system: Tuptime is compatible with OSX, but the integration with LaunchD is not complete. Please, read misc/osx-notes.txt Note about Python 2.7: Tuptime <= 3.5.0 can be executed with Python 2.7. Set the shebang line to '#!/usr/bin/env python' to get it. =========================== | Command line parameters | =========================== These are the command line options, no configuration file is used: -h, --help show this help message and exit -A STARTUP, --at STARTUP limit to this startup number -b, --bootid show boot identifier -c, --csv csv output -d DATETIME_FMT, --date DATETIME_FMT datetime/timestamp format output -e DECIMALS, --dec DECIMALS number of decimals in percentages -E, --exclude STARTUP startup numbers to exclude -f FILE, --filedb FILE database file (/var/lib/tuptime/tuptime.db) -g, --graceful register a graceful shutdown -i, --invert startup number in reverse count | swich between longest/shortest on default output -k, --kernel show kernel version -l, --list enumerate system life as list -n, --noup avoid update values into DB -o TYPE, --order TYPE order enumerate by [u|r|s|e|d|k] -p, --power show power states run + sleep -q, --quiet update values into DB without output -r, --reverse reverse order in listings -s, --seconds output time in seconds and epoch -S STARTUP, --since STARTUP limit from this startup number -t, --table enumerate system life as table --tat TIMESTAMP system status at epoch timestamp --tsince TIMESTAMP limit from this epoch timestamp --tuntil TIMESTAMP limit until this epoch timestamp -U STARTUP, --until STARTUP limit up until this startup number -v, --verbose verbose output -V, --version show version -A , --at Limit values only to this startup number. Take it from (-t) table or (-l) list reports. Negative values are allowed. Examples: tuptime -A 34 tuptime --at 22 tuptime -l -A -1 # List last register -b, --bootid Add boot identifier to the output. Examples: tuptime -b tuptime -t --bootid -h, --help Print a quick reference of the command line parameters. Examples: tuptime -h tuptime --help -c, --csv Report in csv format. Examples: tuptime --csv tuptime --csv -t -d , --date= Change the datetime/timestamp format. By default the output use the configured system locales. The following is a list of all the format codes that the 1989 C standard requires, and these work on all platforms with a standard C implementation. %a Weekday as locale’s abbreviated name. %A Weekday as locale’s full name. %w Weekday as a decimal number, where 0 is Sunday and 6 is Saturday. %d Day of the month as a zero-padded decimal number. %b Month as locale’s abbreviated name. %B Month as locale’s full name. %m Month as a zero-padded decimal number. %y Year without century as a zero-padded decimal number. %Y Year with century as a decimal number. %H Hour (24-hour clock) as a zero-padded decimal number. %I Hour (12-hour clock) as a zero-padded decimal number. %p Locale’s equivalent of either AM or PM. %M Minute as a zero-padded decimal number. %S Second as a zero-padded decimal number. %f Microsecond as a decimal number, zero-padded on the left. %z UTC offset in the form +HHMM or -HHMM (empty string if the the object is naive). %Z Time zone name (empty string if the object is naive). %j Day of the year as a zero-padded decimal number. %U Week number of the year (Sunday as the first day of the week) as a zero padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0. %W Week number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0. %c Locale’s appropriate date and time representation. %x Locale’s appropriate date representation. %X Locale’s appropriate time representation. %% A literal '%' character. To see the full set of format codes supported on your platform, consult the strftime(3) documentation. Examples: tuptime -d '%X %x' # (locale by default) tuptime -d '%H:%M:%S %m-%d-%Y' # MDY style tuptime -d '%H:%M:%S %d-%b-%Y' # DMY style -e DECIMALS, --dec DECIMALS Change the decimal length in percentages. The number is rounded to this value. Examples: tuptime --dec 9 tuptime --power --dec 9 -E , --exclude Avoid take into account the startup numbers defined. Multiple values must be separated by comma ',' and ranges with dash '-'. Default output prints 'trimmed' if the excluded numbers are in middle of the range. Examples: tuptime -E 9 tuptime -E '9-20' tuptime --exclude '2,10-20,50' -f , --filedb= Define an alternative database file. Default is located in '/var/lib/tuptime/tuptime.db'. It takes precedence over environmental variable 'TUPTIME_DBF'. Examples: tuptime -f /var/lib/tuptime/tuptime.db # (Default) tuptime -f /tmp/test1.db tuptime --filedb /tmp/test2.db -g, --graceful Register the time in DB as a graceful shutdown. This option is the way that tuptime have for know if is a good or a bad shutdown. This is used in the init.d and systemd files at the shutdown process. Examples: tuptime -g tuptime --graceful -i, --invert Print startup number in reverse count starting since last boot when it's used in conjunction with (-t), (-l), or (--tat) options. Equal than the output of 'journalctl --list-boots'. Change Longest to Shortest values on default output report. Change Total Uptime to Downtime on (-m) option. Examples: tuptime -t -i tuptime -tib tuptime -l --invert tuptime -i -k, --kernel Add kernel information to the output. Examples: tuptime -k tuptime -lk tuptime --kernel tuptime -t --kernel -l, --list Enumerate system life as a list. Examples: tuptime -l tuptime --list -n, --noup Avoid update values in the DB with the current btime, uptime, etc. Useful when reporting modified DB files. Examples: tuptime -n tuptime --noup -o [u|r|s|e|d|k], --order=[u|r|s|e|d|k] Order enumerate output from table or list by: uptime runtime sleep time end status downtime kernel Therefore, only works in conjunction with (-t) or (-l) options. Examples: tuptime -t -o e tuptime -l -o k tuptime -t --order u tuptime -l --order d -p, --power Print the running and sleeping accumulated time for the correspondent uptime slot. Note that only Python >= 3.6 can register this values. Examples: tuptime -p tuptime --power tuptime -tp tuptime -l --power -q, --quiet Update values into database without compute and print output values. Examples: tuptime -q tuptime --quiet -r, --reverse Reverse order for table or list output. Examples: tuptime -r tuptime -lur tuptime -teo --reverse tuptime -l --reverse -s, --seconds Change default human readable datetime/timestamp style and print times in seconds and datetimes in epoch. Examples: tuptime -s tuptime --seconds -S , --since Limit values only from this startup number. Take it from (-t) table or (-l) list reports. Negative values are allowed. Examples: tuptime -S 34 tuptime --since 22 tuptime -l -S -10 -t, --table Enumerate system life as a table. Examples: tuptime -t tuptime --table --tat Report system status at specific timestamp, register number enclosed and time elapsed and remaining in that matched status. Examples: tuptime --tat 1555280149 tuptime --tat `date -d "1 day ago" +%s` tuptime -cs --tat 1555281311 --tsince Limit report from the given epoch timestamp. Negative values are allowed. Options (-S) and (-U) have precedence over this one. Examples: tuptime --tsince 1454998872 tuptime --tsince `date -d "20-JAN-18" +%s` tuptime --tsince -31536000 # Since one year ago --tuntil Limit report until the given epoch timestamp. Negative values are allowed. Options (-S) and (-U) have precedence over this one. Examples: tuptime --tuntil 1454999619 tuptime --tuntil `date -d "2018-01-20 16:21:42" +%s` tuptime --tuntil -2592000 # Until one month ago -U , --until Limit values only up until this startup number. Take it from (-t) table or (-l) list reports. Negative values are allowed. Examples: tuptime -U 45 tuptime --until 11 tuptime -l -U -10 -v, --verbose Print information about the internals of tuptime. It's good for debugging how it gets the variables. Examples: tuptime -v tuptime --verbose -V, --version: Print version number and exit. Examples: tuptime -V tuptime --version ========================= | Environment variables | ========================= List of supported environmental variables: TUPTIME_DBF Set an alternative database file path. The argument -f, --filedb takes precedence over this. Example for ksh / sh / bash: export TUPTIME_DBF="/opt/tuptime.db" Example for csh / tcsh: setenv TUPTIME_DBF /opt/tuptime.db Example for systemd unit: Environment="TUPTIME_DBF='/opt/tuptime.db'" ====================== | Schedule execution | ====================== It's important to have a schedule execution. If the init or systemd scripts are installed as it needs, the program only update values at startup and shutdown, but if the system fails, hangs or whatever, the uptime time will be lost. Be sure that the cron entry or, in other case, the systemd tuptime-sync.[timer|service] units are installed as expected. Why is it needed the use of an other .service file with the name tuptime-sync? The default tuptime.service file uses the option 'RemainAfterExit=true' and nowadays the systemd timer can't restart an already running service, so, this is the best workaround. By default, both have a 5 minutes scheduled execution. Feel free to lower that value if your requirements are narrow. ================== | Default output | ================== System startups: Total number of system startups registered since first timestamp available. System shutdowns: Total number of shutdowns done correctly or incorrectly. System life: Time counter since first startup timestamp available. System uptime: System downtime: Percentage of time and time counter. Longest uptime: Longest downtime: Time counter and date with the longest/shortest uptime register. Average uptime: Average downtime: Average time counter. Current uptime: Actual time counter and datetime since registered boot timestamp. ====================================== | DB format changes between versions | ====================================== From 2.x to 3.x: From 2.x to 3.0, reorder columns, added offbtime, endst and downtime. From 3.0 to 3.1, added kernel column. The migration scripts are loated at: https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-2.0-to-3.0.sh https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.0-to-3.1.sh From 3.x to 4.x: Added rntime and slptime registers, also uptime, rntime, slptime, downtime are now integers. The migrations is done automatically from Tuptime. Also, the migration script is located at: https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.1-to-4.0.sh From 4.x to 5.x: Added bootid register. The migrations is done automatically from Tuptime. Also, the migration script is located at: https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-4.0-to-5.0.sh ================== | Database Specs | ================== Tuptime use a sqlite3 database located in '/var/lib/tuptime/tuptime.db' with the following format: tuptime (bootid text, btime integer, uptime integer, rntime integer, slptime integer, offbtime integer, endst integer, downtime integer, kernel text) bootid Unique boot identifier (if it exists) btime Startup timestamp in epoch uptime Uptime seconds rntime Running time seconds slptime Sleeping time seconds offbtime Shutdown timestamp in epoch endst Type of shutdown [1 ok | 0 bad] downtime Downtime seconds kernel Name of the kernel The number of startup sequency is extracted from 'rowid', the signed integer key that uniquely identifies the row within its table in sqlite. For reset all the values, simply delete the database file. It is possible to query and modify it directly, but it's better to use any of the scripts located under 'misc/scripts' for modify, join or check it. If a complete row is deleted and the database is not recreate completely (vacuum), rowid keep the real information about startup number. Tuptime will advert about it if the verbose mode is enabled. ============================ | About sync date and time | ============================ Large jumps over the date and time in a system can cause problems, especially when the uptime value is low. Take care of the hardware clock RTC (if the system have), or how the operating system initializes their clock at startup. Tuptime register the timestamp reported by the system at first execution after boot and use it to calculate the previous shutdown time and the current uptime value. The command 'who -b' do more less the same. If the system starts with a incorrect datetime and some external time sync is used, like ntpd or timesyncd, a few time after boot the startup datetime reported by the system may change. This behaviour is also related with adjustments performed by adjtime(3), systems running inside virtualized environments, servers with high load or with high disk I/O, wrong computation of jiffies / HZ and the problema of lost ticks. A slightly drift is compensate over the uptime value, but a very high drift can cause abnormal registers and output, for example, Raspberry Pi don't have any clock and initializes from 1 January 1970. A downtime of 0, when it's clear than it was physically impossible, it's a common indicator than the clock had a large backward sync jump. As Tuptime have time dependency, their Systemd service unit pulls time-sync.target and it's ordered before it. Note that it doesn't assure that the clock is synchronized, only that it needs it. On last releases of Systemd, there is a new unit that delays reaching time-sync.target until stay on sync 'systemd-time-wait-sync',but it isn't enable by default nowadays. A few recommendations / workarounds are: Enable 'systemd-time-wait-sync'. But take care because it can delay the execution more than expected and a startup can be lost. Check open issues on Systemd's GitHub. Use 'systemd-timesyncd' if it is available (timedatectl set-ntp true), it can also keep the time in systems without hardware clock. Disable any other time sync service like 'ntp', 'chrony', 'openntpd' to avoid issues, most related with the use in conjunction with 'systemd-time-wait-sync'. Use any time sync service like 'ntp', 'chrony' or 'openntpd' and disable 'systemd-timesyncd'. Add the time sync service requirement on 'tuptime.service' at the end of 'After=' and 'Wants=' lines. Force a time sync in the 'tuptime.unit' file. Add under '[Service]' definition the line 'ExecStartPre=/usr/bin/ntpdate pool.ntp.org' If the systems is a single-board computer, like Raspberry Pi or Beaglebone, use a RTC module and configure it properly. Without Systemd, use a ntp client and delay the execution of tuptime until the time is synced. Also, 'ntp-wait' can be added to the init script and the cron line preceeding the tuptime execution. Try and failure is the best way to find the right solution for the environment in which the system is running. The drift value that Tuptime have reference is reported in verbose mode 'tuptime -v' in the line 'INFO:Drift over btime'. Values around +-1 are common and can be considered as normal. Systemd timer issues: https://github.com/systemd/systemd/issues/14061 https://github.com/systemd/systemd/issues/8683 https://github.com/systemd/systemd/issues/5097 Good reads: https://tools.ietf.org/html/rfc1589 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=119971 http://man7.org/linux/man-pages/man2/clock_gettime.2.html https://unix.stackexchange.com/questions/118631/how-can-i-measure-and-prevent-clock-drift Timekeeping in VMware Virtual Machines (Document) https://venam.nixers.net/blog/unix/2020/05/02/time-on-unix.html https://papers.freebsd.org/2002/phk-timecounters.files/timecounter.pdf ======================== | Date and Epoch notes | ======================== The options --tsince, --tuntil and --tat use epoch timestamp format, as quick reference here are some notes: Seconds equivalences: 1 year 31536000 seconds 1 month 2592000 seconds 1 week 604800 seconds 1 day 86400 seconds 1 hour 3600 seconds Convert human datetime to epoch: date -d "JAN-18-2018" +%s date -d "20-JAN-18" +%s date -d "2018-01-20 16:21:42" +%s date -d "2018-01-20 16:21:42 GMT+2" +%s date -d "3 week ago" +%s date -d "1 year ago" +%s date -d "now" +%s Convert epoch to human datetime: date -d "@1516748400" Preceding conversions uses GNU date, on FreeBSD install "coreutils" and call them with "gdate". More info about datetime input formats: man date 'https://www.gnu.org/software/coreutils/manual/html_node/ Date-input-formats.html#Date-input-formats' =========== | Caveats | =========== Tuptime can be executed with 'root' user, but it's recommendable the use of an unprivileged user, like '_tuptime' as the installation section points to avoid security problems. In any case, if the database file is not writable by the current execution user, the changes are not saved. This situation is covered for print values, but not for update database, that's why a user who can write in the database file must execute tuptime, usually from the init manager, to update values at startup and shutdown, at least. Kernel name is registered each time that tuptime update values. If ksplice is used to update the kernel, last kernel will have the whole time assignment since the last system start. Tuptime is backed by a SQLite database. It is highly resilient but fails can occur in any software and hardware level, producing a database corruption. Here is an excellent documentation about it https://www.sqlite.org/howtocorrupt.html If a backup of the DB file is needed in other place, take a look to Litestream. This is a streaming replication tool for SQLite databases, it works fine with Tuptime. https://litestream.io/ The uptime reference is linked with the wall clock time elapsed since boot, not the running effective time. Running (effective time) and hibernate power states are childs of the uptime and report accumulated values inside their time range, which counts since the startup until the shutdown. If the system have configured a time zone with daylight saving time (DST), to avoid problems when the time change, the real time clock (RTC) should have set to universal time coordinated (UTC), not local time. Check it with 'timedatectl' command. Reports involving DST changes are completely right. The system current local time zone is the reference for all timestamps printed. https://tldp.org/HOWTO/Clock-2.html If the Linux system execute a full force shutdown or reboot, like for example "systemctl --force --force reboot", as it doesn't follow the steps to produce a clean shutdown, the "End status" remains in bad state. The name of the tool is based on the contraction of the words Total Uptime.