1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
.. _writingrules:
Adding a New Rule Type
======================
This document describes how to create a new rule type. Built in rule types live in ``elastalert/ruletypes.py``
and are subclasses of ``RuleType``. At the minimum, your rule needs to implement ``add_data``.
Your class may implement several functions from ``RuleType``:
.. code-block:: python
class AwesomeNewRule(RuleType):
# ...
def add_data(self, data):
# ...
def get_match_str(self, match):
# ...
def garbage_collect(self, timestamp):
# ...
You can import new rule types by specifying the type as ``module.file.RuleName``, where module is the name of a Python module, or folder
containing ``__init__.py``, and file is the name of the Python file containing a ``RuleType`` subclass named ``RuleName``.
Basics
------
The ``RuleType`` instance remains in memory while ElastAlert is running, receives data, keeps track of its state,
and generates matches. Several important member properties are created in the ``__init__`` method of ``RuleType``:
``self.rules``: This dictionary is loaded from the rule configuration file. If there is a ``timeframe`` configuration
option, this will be automatically converted to a ``datetime.timedelta`` object when the rules are loaded.
``self.matches``: This is where ElastAlert checks for matches from the rule. Whatever information is relevant to the match
(generally coming from the fields in Elasticsearch) should be put into a dictionary object and
added to ``self.matches``. ElastAlert will pop items out periodically and send alerts based on these objects. It is
recommended that you use ``self.add_match(match)`` to add matches. In addition to appending to ``self.matches``,
``self.add_match`` will convert the datetime ``@timestamp`` back into an ISO8601 timestamp.
``self.required_options``: This is a set of options that must exist in the configuration file. ElastAlert will
ensure that all of these fields exist before trying to instantiate a ``RuleType`` instance.
add_data(self, data):
---------------------
When ElastAlert queries Elasticsearch, it will pass all of the hits to the rule type by calling ``add_data``.
``data`` is a list of dictionary objects which contain all of the fields in ``include``, ``query_key`` and ``compare_key``
if they exist, and ``@timestamp`` as a datetime object. They will always come in chronological order sorted by '@timestamp'.
get_match_str(self, match):
---------------------------
Alerts will call this function to get a human readable string about a match for an alert. Match will be the same
object that was added to ``self.matches``, and ``rules`` the same as ``self.rules``. The ``RuleType`` base implementation
will return an empty string. Note that by default, the alert text will already contain the key-value pairs from the match. This
should return a string that gives some information about the match in the context of this specific RuleType.
garbage_collect(self, timestamp):
---------------------------------
This will be called after ElastAlert has run over a time period ending in ``timestamp`` and should be used
to clear any state that may be obsolete as of ``timestamp``. ``timestamp`` is a datetime object.
Tutorial
--------
As an example, we are going to create a rule type for detecting suspicious logins. Let's imagine the data we are querying is login
events that contains IP address, username and a timestamp. Our configuration will take a list of usernames and a time range
and alert if a login occurs in the time range. First, let's create a modules folder in the base ElastAlert folder:
.. code-block:: console
$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py
Now, in a file named ``my_rules.py``, add
.. code-block:: python
import dateutil.parser
from elastalert.ruletypes import RuleType
# elastalert.util includes useful utility functions
# such as converting from timestamp to datetime obj
from elastalert.util import ts_to_dt
class AwesomeRule(RuleType):
# By setting required_options to a set of strings
# You can ensure that the rule config file specifies all
# of the options. Otherwise, ElastAlert will throw an exception
# when trying to load the rule.
required_options = set(['time_start', 'time_end', 'usernames'])
# add_data will be called each time Elasticsearch is queried.
# data is a list of documents from Elasticsearch, sorted by timestamp,
# including all the fields that the config specifies with "include"
def add_data(self, data):
for document in data:
# To access config options, use self.rules
if document['username'] in self.rules['usernames']:
# Convert the timestamp to a time object
login_time = document['@timestamp'].time()
# Convert time_start and time_end to time objects
time_start = dateutil.parser.parse(self.rules['time_start']).time()
time_end = dateutil.parser.parse(self.rules['time_end']).time()
# If the time falls between start and end
if login_time > time_start and login_time < time_end:
# To add a match, use self.add_match
self.add_match(document)
# The results of get_match_str will appear in the alert text
def get_match_str(self, match):
return "%s logged in between %s and %s" % (match['username'],
self.rules['time_start'],
self.rules['time_end'])
# garbage_collect is called indicating that ElastAlert has already been run up to timestamp
# It is useful for knowing that there were no query results from Elasticsearch because
# add_data will not be called with an empty list
def garbage_collect(self, timestamp):
pass
In the rule configuration file, ``example_rules/example_login_rule.yaml``, we are going to specify this rule by writing
.. code-block:: yaml
name: "Example login rule"
es_host: elasticsearch.example.com
es_port: 14900
type: "elastalert_modules.my_rules.AwesomeRule"
# Alert if admin, userXYZ or foobaz log in between 8 PM and midnight
time_start: "20:00"
time_end: "24:00"
usernames:
- "admin"
- "userXYZ"
- "foobaz"
# We require the username field from documents
include:
- "username"
alert:
- debug
ElastAlert will attempt to import the rule with ``from elastalert_modules.my_rules import AwesomeRule``.
This means that the folder must be in a location where it can be imported as a Python module.
An alert from this rule will look something like::
Example login rule
userXYZ logged in between 20:00 and 24:00
@timestamp: 2015-03-02T22:23:24Z
username: userXYZ
|
