1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
Using Logstash and ElasticSearch to Process Eliot Logs
======================================================
.. note:: Logstash, Elasticsearch and Kibana change frequently. These instructions might not be quite accurate.
`ElasticSearch`_ is a search and analytics engine which can be used to store Eliot logging output.
The logs can then be browsed by humans using the `Kibana`_ web UI, or on the command-line using the `logstash-cli`_ tool.
Automated systems can access the logs using the ElasticSearch query API.
`Logstash`_ is a log processing tool that can be used to load Eliot log files into ElasticSearch.
The combination of ElasticSearch, Logstash, and Kibana is sometimes referred to as ELK.
.. _logstash-cli: https://github.com/jedi4ever/logstash-cli
.. _Logstash: http://logstash.net/
.. _ElasticSearch: http://elasticsearch.org
.. _Kibana: http://www.elasticsearch.org/overview/kibana/
Example Logstash Configuration
------------------------------
Assuming each Eliot message is written out as a JSON message on its own line (which is the case for ``eliot.to_file()`` and ``eliot.logwriter.ThreadedFileWriter``), the following Logstash configuration will load these log messages into an in-process ElasticSearch database:
:download:`logstash_standalone.conf`
.. literalinclude:: logstash_standalone.conf
We can then pipe JSON messages from Eliot into ElasticSearch using Logstash:
.. code-block:: console
$ python examples/stdout.py | logstash web -- agent --config logstash_standalone.conf
You can then use the Kibana UI to search and browse the logs by visiting http://localhost:9292/.
|
