1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
logdata-anomaly-miner (0.0.7-1) unstable; urgency=low
The datetime parsing DateTimeModelElement was reimplemented
to fix various shortcomings of strptime in Python and libc.
This will require changes in configuration due to API changes,
e.g.:
-timeModel=DateTimeModelElement('time', '%b %d %H:%M:%S', 15, False)
+timeModel=DateTimeModelElement('time', '%b %d %H:%M:%S')
See /usr/lib/logdata-anomaly-miner/aminer/parsing/DateTimeModelElement.py
source code documentation for currently supported datetime format
options.
The code for reading log input was improved to allow also input
from UNIX sockets. Thus the configuration was changed to support
those modes:
-configProperties['LogFileList']=['/var/log/auth.log', ...
+configProperties['LogResourceList'] = ['file:///var/log/auth.log', ...
-- Roman Fiedler <roman.fiedler@ait.ac.at> Mon, 9 Jan 2017 18:00:00 +0000
logdata-anomaly-miner (0.0.6-1) unstable; urgency=low
The input IO-handling was redesigned, thus introducing following
API changes. The changes are flaged with (D)eveloper and (U)ser
to indicate if only developers of own AMiner addons are affected
or also users may need to migrate their configuration.
* Upper layers receive LogAtom objects instead of log lines,
parsing data as separate parameters. Thus also separate paths
for forwarding of parsed and unparsed atoms are not required
any more. See below for details (D, U):
* Update any own UnparsedAtomHandler/ParsedAtomHandlerInterface
implementations to use new interface "input.AtomHandlerInterface"
and access to additional information to new methods and
fields (D):
-from aminer.parsing import ParsedAtomHandlerInterface
+from aminer.input import AtomHandlerInterface
-class YourHandler(ParsedAtomHandlerInterface, ...
+class YourHandler(AtomHandlerInterface,
- def receiveParsedAtom(self, atomData, parserMatch):
+ def receiveAtom(self, logAtom):
- timestamp=parserMatch.getDefaultTimestamp()
+ timestamp=logAtom.getTimestamp()
+ parserMatch=logAtom.parserMatch
- print '%s' % atomData
+ print '%s' % logAtom.rawData
* With parsed/unparsed atom processing path convergence, naming
of other classes does not make sense any more (U):
-from aminer.analysis import VolatileLogarithmicBackoffParsedAtomHistory
+from aminer.util import VolatileLogarithmicBackoffAtomHistory
- from aminer.analysis import ParsedAtomFilters
+ from aminer.analysis import AtomFilters
- matchAction=Rules.ParsedAtomFilterMatchAction(...
+ matchAction=Rules.AtomFilterMatchAction(...
- parsedAtomHandlers=[]
- unparsedAtomHandlers=[]
- analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory(
- parsingModel, parsedAtomHandlers, unparsedAtomHandlers, ...
+ atomFilter=AtomFilters.SubhandlerFilter(None)
+ analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory(
+ parsingModel, [atomFilter], ...
For handling of unparsed atoms:
- unparsedAtomHandlers.append(SimpleUnparsedAtomHandler(anomalyEventHandlers))
+ atomFilter.addHandler(SimpleUnparsedAtomHandler(anomalyEventHandlers),
+ stopWhenHandledFlag=True)
For handling of parsed atoms:
- parsedAtomHandlers.append(...
+ atomFilter.addHandler(...
-- Roman Fiedler <roman.fiedler@ait.ac.at> Fri, 4 Nov 2016 18:00:00 +0000
logdata-anomaly-miner (0.0.5-1) unstable; urgency=low
Following API changes were introduced:
* Lower input layers dealing with binary data stream reading,
splitting into atoms and forwarding data to the parsing model
were redesigned. Following configuration changes are required
to adapt "config.py" and probably "analysis.py" to the new
API:
* analysisContext.registerComponent(): registerAsRawAtomHandler
parameter not needed any more, can be removed.
* SimpleParsingModelRawAtomHandler is not needed any more,
that part can be replaced by configuration:
# Now define the AtomizerFactory using the model. A simple line
# based one is usually sufficient.
from aminer.input import SimpleByteStreamLineAtomizerFactory
analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory(
parsingModel, parsedAtomHandlers, unparsedAtomHandlers,
anomalyEventHandlers, defaultTimestampPath='/model/syslog/time')
* SimpleUnparsedAtomHandler was moved from "aminer.events"
to "aminer.input".
-- Roman Fiedler <roman.fiedler@ait.ac.at> Mon, 11 Oct 2016 18:00:00 +0000
logdata-anomaly-miner (0.0.4-1) unstable; urgency=low
Following API changes were introduced:
* Event handling (general): Change of EventHandlerInterface
to include also eventSource as last parameter. See
/usr/lib/logdata-anomaly-miner/aminer/events/__init__.py
* VolatileLogarithmicBackoffEventHistory: Added event ID and
source to stored tuple to allow unique identification of events.
Split result of "getHistory()" to include "eventId, eventType,
eventMessage, sortedLogLines, eventData, eventSource".
-- Roman Fiedler <roman.fiedler@ait.ac.at> Fri, 26 Aug 2016 15:15:00 +0000
logdata-anomaly-miner (0.0.3-1) unstable; urgency=low
Following API changes were introduced:
* To improve readability of configuration files, main parser,
analysis and event classes were added to the submodule namespaces.
After imports directly from the submodule, e.g.
"from aminer.parsing import FixedDataModelElement",
the name duplication "FixedDataModelElement.FixedDataModelElement"
is not required any more, "FixedDataModelElement" is sufficient.
Use "sed -i -e 's/Name.Name/Name/g' [files]" to adapt.
* Component timing was restructured to allow forensic/realtime
triggering. Therefore also clean interface was added, which
is now also used to reduce redundant code in component registration.
Old way:
analysisContext.registerComponent(newMatchPathDetector,
componentName=None, registerAsRawAtomHandler=False,
registerAsTimeTriggeredHandler=True)
New way:
analysisContext.registerComponent(newMatchPathDetector,
registerAsRawAtomHandler=False)
For own custom time-triggered components, make sure to implement
the "aminer.util.TimeTriggeredComponentInterface". Use any standard
component, e.g. "/usr/lib/logdata-anomaly-miner/aminer/analysis/NewMatchPathDetector.py"
as example.
* Introduction of "AnalysisContext" to have common handle for
all data required to perform the analysis. Therefore also
the signature of "buildAnalysisPipeline" in "config.py/analysis.py"
has changed from
def buildAnalysisPipeline(aminerConfig):
to
def buildAnalysisPipeline(analysisContext):
Old references to "aminerConfig" within the configuration
script have to be replaced by "analysisContext.aminerConfig".
-- Roman Fiedler <roman.fiedler@ait.ac.at> Thu, 21 Jul 2016 19:00:00 +0000
|
