1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
# Introduction
mtail is intended to run one per machine, and serve as monitoring glue for multiple applications running on that machine. It runs one or more programs in a 1:1 mapping to those client applications.
## Configuration Overview
mtail is configured through commandline flags.
The `--help` flag will print a list of flags for configuring `mtail`.
(Flags may be prefixed with either `-` or `--`)
Basic flags necessary to start `mtail`:
* `--logs` is a comma separated list of filenames to extract from, but can also be used multiple times, and each filename can be a [glob pattern](http://godoc.org/path/filepath#Match). Named pipes can be read from when passed as a filename to this flag.
* `--progs` is a directory path containing [mtail programs](Language.md). Programs must have the `.mtail` suffix.
mtail runs an HTTP server on port 3903, which can be changed with the `--port` flag.
# Details
## Launching mtail
```
mtail --progs /etc/mtail --logs /var/log/syslog --logs /var/log/ntp/peerstats
```
`mtail` will start to read the specified logs from their current end-of-file,
and read new updates appended to these logs as they arrive. It will attempt to
correctly handle log files that have been rotated by renaming or symlink
changes.
### Getting the logs in
Use `--logs` multiple times to pass in glob patterns that match the logs you
want to tail. This includes named pipes.
### Polling the file system
If your system is not supported by `fsnotify` then mtail will fall back to polling mode. You can also specify this explicitly with the `--poll_interval` flag, for example
```
mtail --progs /etc/mtail --logs /var/log/syslog --poll_interval 250ms
```
*N.B.* `mtail` will only discover new files if using `fsnotify`. If you rely on polling mode, then only the files that mtail sees at program startup will be watched.
### Disabling `fsnotify`
In some cases, the log watcher can not process update events from the kernel fast enough and you may see
```
fsnotify error: fsnotify queue overflow
```
errors in the `mtail` log. This will also manifest as counters not updating anymore.
If your incoming log rate is high enough to trigger this condition, try forcing mtail to only use polling mode by adding the flag `--disable_fsnotify`.
When fsnotify is disabled, new mtail programs and new log files won't be detected after program startup.
### Launching under Docker
`mtail` can be run as a sidecar process if you expose an application container's logs with a volume.
`docker run -d --name myapp -v /var/log/myapp myapp`
for example exports a volume called `/var/log/myapp` (named the same as the
hypothetical path where `myapp`s logs are written.
Then launch the `mtail` docker image and pass in the volume:
docker run -dP \
--name myapp-mtail \
--volumes-from myapp \
-v examples:/etc/mtail \
mtail --logs /var/log/myapp --progs /etc/mtail
This example fetches the volumes from the `myapp` container, and mounts them in
the mtail container (which we've called `myapp-mtail`). We also mount the
`examples` directory as `/etc/mtail` in the container. We launch `mtail` with
the `logs` and `progs` flags to point to our two mounted volumes.
The `-P` flag ensures `mtail-myapp`'s port 3903 is exposed for collection,
refer to `docker ps` to find out where it's mapped to on the host.
## Writing the programme
Read the [Programming Guide](Programming-Guide.md) for instructions on how to write an `mtail` program.
## Getting the Metrics Out
### Pull based collection
Point your collection tool at `localhost:3903/json` for JSON format metrics.
Prometheus can be directed to the /metrics endpoint for Prometheus text-based format.
### Push based collection
Use the `collectd_socketpath` or `graphite_host_port` flags to enable pushing to a collectd or graphite instance.
Configure collectd on the same machine to use the unixsock plugin, and set `collectd_socketpath` to that unix socket.
```
mtail --progs /etc/mtail --logs /var/log/syslog,/var/log/rsyncd.log --collectd_socketpath=/var/run/collectd-unixsock
```
Set `graphite_host_port` to be the host:port of the carbon server.
```
mtail --progs /etc/mtail --logs /var/log/syslog,/var/log/rsyncd.log --graphite_host_port=localhost:9999
```
Likewise, set `statsd_hostport` to the host:port of the statsd server.
Additionally, the flag `metric_push_interval_seconds` can be used to configure the push frequency. It defaults to 60, i.e. a push every minute.
## Setting a default timezone
The `--override_timezone` flag sets the timezone that `mtail` uses for timestamp conversion. By default, `mtail` assumes timestamps are in UTC.
To use the machine's local timezone, `--override_timezone=Local` can be used.
## Troubleshooting
Lots of state is logged to the log file, by default in `/tmp/mtail.INFO`. See [Troubleshooting](Troubleshooting.md) for more information.
N.B. Oneshot mode (the `one_shot` flag on the commandline) can be used to check
that a program is correctly reading metrics from a log, but with the following
caveats:
* Unlike normal operations, oneshot mode will read the logs from the start of
the file to the end, then close them -- it does not continuously tail the
file
* The metrics will be printed to standard out when the logs are finished being
read from.
* mtail will exit after the metrics are printed out.
This mode is useful for debugging the behaviour of `mtail` programs and
possibly for permissions checking.
|
