File: changelog

package info (click to toggle)
logdata-anomaly-miner 2.2.2-1
  • links: PTS, VCS
  • area: main
  • in suites: bookworm, bullseye, sid
  • size: 4,464 kB
  • sloc: python: 24,066; sh: 1,860; xml: 821; makefile: 19
file content (338 lines) | stat: -rw-r--r-- 13,146 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
logdata-anomaly-miner (2.2.1) unstable; urgency=low
  Bugfixes:
  * Fixed warnigs due to files in Persistency-Directory
  * Fixed ACL-problems in dockerfile and autocreate /var/lib/aminer/log
  Changes:
  * added simple test for dockercontainer
  * negate result of the timeout-command. 1 is okay. 0 must be an error
  * added bullseye-tests
  * make tmp-dir in debian-bullseye-test and debian-buster-test unique

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Mon,  25 Jan 2021 12:00:00 +0000

logdata-anomaly-miner (2.2.0) unstable; urgency=low

  Changes:
  * Added Dockerfile
  * Addes checks for acl of persistency directory
  * Added VariableCorrelationDetector
  * Added tool for managing multiple persistency files
  * Added supress-list for output
  * Added suspend-mode to remote-control
  * Added requirements.txt
  * Extended documentation
  * Extended yaml-configuration-support
  * Standardize command line parameters
  * Removed --Forground cli parameter
  * Fixed Security warnings by removing functions that allow race-condition
  * Refactoring
  * Ethical correct naming of variables
  * Enhanced testing
  * Added statistic outputs
  * Enhanced status info output
  * Changed global learn_mode behavior
  * Added RemoteControlSocket to yaml-config
  * Reimplemented the default mailnotificationhandler
  Bugfixes:
  * Fixed typos in documentation
  * Fixed issue with the AtomFilter in the yaml-config
  * Fixed order of ETD in yaml-config
  * Fixed various issues in persistency

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  18 Dec 2020 17:00:00 +0000

logdata-anomaly-miner (2.1.0) unstable; urgency=low

  Changes:
  * Added VariableTypeDetector,EventTypeDetector and EventCorrelationDetector
  * Added support for unclean format strings in the DateTimeModelElement
  * Added timezones to the DateTimeModelElement
  * Enhanced ApacheAccessModel
  * Yamlconfig: added support for kafka stream
  * Removed cpu limit configuration
  * Various refactoring
  * Yamlconfig: added support for more detectors
  * Added new command-line-parameters
  * Renamed executables to aminer.py and aminerremotecontroly.py
  * Run Aminer in forgroundd-mode per default
  * Added various unit-tests
  * Improved yamlconfig and checks
  * Added start-config for parser to yamlconfig
  * Renamed config templates
  * Removed imports from init.py for better modularity
  * Created AnalysisComponentsPerformanceTests for the EventTypeDetector
  * Extended demo-config
  * Renamed whitelist to allowlist
  * Added warnings for non-existent resources
  * Changed default of auto_include_flag to false
  Bugfixes:
  * Fixed some exit() in forks
  * Fixed debian files
  * Fixed JSON output of the AffectedLogAtomValues in all detectors
  * Fixed normal output of the NewMatchPathValueDetector
  * Fixed reoccuring alerting in MissingMatchPathValueDetector

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Thu,  05 Nov 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.2) unstable; urgency=low

  Changes:
  * Added help parameters
  * Added help-screen
  * Added version parameter
  * Adden path and value filter
  * Change time model of ApacheAccessModel for arbitrary time zones
  * Update link to documentation
  * Added SECURITY.md
  * Refactoring
  * Updated man-page
  * Added unit-tests for loadYamlconfig
  Bugfixes:
  * Fixed header comment type in schema file
  * Fix debian files

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Wed,  17 Jul 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.1) unstable; urgency=low

  Changes:
  * Updated documentation
  * Updated testcases
  * Updated demos
  * Updated debian files
  * Added copyright headers
  * Added executable bit to AMiner
  
 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Wed,  24 Jun 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.0) bionic; urgency=low

  Changes:
  * Updated documentation
  * Added functions getNameByComponent and getIdByComponent to
    AnalysisChild.py
  * Update DefaultMailNotificationEventHandler.py to python3
  * Extended AMinerRemoteControl
  * Added support for configuration in yaml format
  * Refactoring
  * Added KafkaEventHandler
  * Added JsonConverterHandler
  * Added NewMatchIdValueComboDetector
  * Enabled multiple default timestamp paths
  * Added debug feature ParserCount
  * Added unit and integration tests
  * Added installer script
  * Added VerboseUnparsedHandler
  Bugfixes including:
  * Fixed dependencies in Debian packaging
  * Fixed typo in various analysis components
  * Fixed import of ModelElementInterface in various parsing components
  * Fixed issues with byte/string comparison
  * Fixed issue in DecimalIntegerValueModelElement, when parsing
    integer including sign and padding character
  * Fixed unnecessary long blocking time in SimpleMultisourceAtomSync
  * Changed minum matchLen in DelimitedDataModelElement to 1 byte
  * Fixed timezone offset in ModuloTimeMatchRule
  * Minor bugfixes

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  29 May 2020 17:00:00 +0000

logdata-anomaly-miner (1.0.0) bionic; urgency=low
  
  Changes:
  * Ported code to Python 3
  * Code cleanup using pylint
  * Added util/JsonUtil.py to encode byte strings for storing them as json objects
  * Added docs/development-procedures.txt which documents development procedures
  Features:
  * New MissingMatchPathListValueDetector to detect stream interuption
  * Added parsing support for kernel IP layer martian package messages
  * Systemd parsing of apt invocation messages.
  Bugfixes:
  * AnalysisChild: handle remote control client connection errors correctly
  * Various bugfixes

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Tue,  2 Oct 2018 17:00:00 +0000

logdata-anomaly-miner (0.0.8) xenial; urgency=low

  Apart from bugfixes, new parsing and analysis components were added:
  * Base64StringModelElement
  * DecimalFloatValueModelElement
  * StringRegexMatchRule
  * EnhancedNewMatchPathValueComboDetector

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Tue, 30 May 2017 17:00:00 +0000

logdata-anomaly-miner (0.0.7) xenial; urgency=low

  The datetime parsing DateTimeModelElement was reimplemented
  to fix various shortcomings of strptime in Python and libc.
  This will require changes in configuration due to API changes,
  e.g.:

  -time_model=DateTimeModelElement('time', '%b %d %H:%M:%S', 15, False)
  +time_model=DateTimeModelElement('time', '%b %d %H:%M:%S')

  See /usr/lib/logdata-anomaly-miner/aminer/parsing/DateTimeModelElement.py
  source code documentation for currently supported datetime format
  options.

  The code for reading log input was improved to allow also input
  from UNIX sockets. Thus the configuration was changed to support
  those modes:

  -config_properties['LogFileList']=['/var/log/auth.log', ...
  +config_properties['LogResourceList'] = ['file:///var/log/auth.log', ...

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Mon, 9 Jan 2017 18:00:00 +0000

logdata-anomaly-miner (0.0.6) xenial; urgency=low

  The input IO-handling was redesigned, thus introducing following
  API changes. The changes are flaged with (D)eveloper and (U)ser
  to indicate if only developers of own AMiner addons are affected
  or also users may need to migrate their configuration.

  * Upper layers receive LogAtom objects instead of log lines,
    parsing data as separate parameters. Thus also separate paths
    for forwarding of parsed and unparsed atoms are not required
    any more. See below for details (D, U):

    * Update any own UnparsedAtomHandler/ParsedAtomHandlerInterface
      implementations to use new interface "input.AtomHandlerInterface"
      and access to additional information to new methods and
      fields (D):

  -from aminer.parsing import ParsedAtomHandlerInterface
  +from aminer.input import AtomHandlerInterface
  -class YourHandler(ParsedAtomHandlerInterface, ...
  +class YourHandler(AtomHandlerInterface,
  -  def receiveParsedAtom(self, atom_data, parser_match):
  +  def receive_atom(self, log_atom):
  -    timestamp=parser_match.get_default_timestamp()
  +    timestamp=log_atom.get_timestamp()
  +    parser_match=log_atom.parser_match
  -    print '%s' % atom_data
  +    print '%s' % log_atom.rawData

    * With parsed/unparsed atom processing path convergence, naming
      of other classes does not make sense any more (U):

  -from aminer.analysis import VolatileLogarithmicBackoffParsedAtomHistory
  +from aminer.util import VolatileLogarithmicBackoffAtomHistory
  - from aminer.analysis import ParsedAtomFilters
  + from aminer.analysis import AtomFilters
  - match_action=Rules.ParsedAtomFilterMatchAction(...
  + match_action=Rules.AtomFilterMatchAction(...

  - parsed_atom_handlers=[]
  - unparsed_atom_handlers=[]
  - analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
  -     parsing_model, parsed_atom_handlers, unparsed_atom_handlers, ...
  + atom_filter=AtomFilters.SubhandlerFilter(None)
  + analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
  +     parsing_model, [atom_filter], ...

  For handling of unparsed atoms:

  - unparsed_atom_handlers.append(SimpleUnparsedAtomHandler(anomaly_event_handlers))
  + atom_filter.add_handler(SimpleUnparsedAtomHandler(anomaly_event_handlers),
  +     stop_when_handled_flag=True)

  For handling of parsed atoms:

  - parsed_atom_handlers.append(...
  + atom_filter.add_handler(...

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Fri, 4 Nov 2016 18:00:00 +0000

logdata-anomaly-miner (0.0.5) xenial; urgency=low

  Following API changes were introduced:

  * Lower input layers dealing with binary data stream reading,
    splitting into atoms and forwarding data to the parsing model
    were redesigned. Following configuration changes are required
    to adapt "config.py" and probably "analysis.py" to the new
    API:

    * analysis_context.register_component(): register_as_raw_atom_handler
      parameter not needed any more, can be removed.

    * SimpleParsingModelRawAtomHandler is not needed any more,
      that part can be replaced by configuration:

  # Now define the AtomizerFactory using the model. A simple line
  # based one is usually sufficient.
    from aminer.input import SimpleByteStreamLineAtomizerFactory
    analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
        parsing_model, parsed_atom_handlers, unparsed_atom_handlers,
        anomaly_event_handlers, default_timestamp_paths=['/model/syslog/time'])

    * SimpleUnparsedAtomHandler was moved from "aminer.events"
      to "aminer.input".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Mon, 11 Oct 2016 18:00:00 +0000

logdata-anomaly-miner (0.0.4) xenial; urgency=low

  Following API changes were introduced:

  * Event handling (general): Change of EventHandlerInterface
    to include also event_source as last parameter. See
    /usr/lib/logdata-anomaly-miner/aminer/events/__init__.py

  * VolatileLogarithmicBackoffEventHistory: Added event ID and
    source to stored tuple to allow unique identification of events.
    Split result of "getHistory()" to include "eventId, eventType,
    event_message, sorted_log_lines, event_data, event_source".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Fri, 26 Aug 2016 15:15:00 +0000

logdata-anomaly-miner (0.0.3) xenial; urgency=low

  Following API changes were introduced:

  * To improve readability of configuration files, main parser,
    analysis and event classes were added to the submodule namespaces.
    After imports directly from the submodule, e.g.
    "from aminer.parsing import FixedDataModelElement",
    the name duplication "FixedDataModelElement.FixedDataModelElement"
    is not required any more, "FixedDataModelElement" is sufficient.
    Use "sed -i -e 's/Name.Name/Name/g' [files]" to adapt.

  * Component timing was restructured to allow forensic/realtime
    triggering. Therefore also clean interface was added, which
    is now also used to reduce redundant code in component registration.
    Old way:

    analysis_context.register_component(new_match_path_detector,
      component_name=None, register_as_raw_atom_handler=False,
      register_as_time_triggered_handler=True)

    New way:

    analysis_context.register_component(new_match_path_detector,
      register_as_raw_atom_handler=False)

    For own custom time-triggered components, make sure to implement
    the "aminer.util.TimeTriggeredComponentInterface". Use any standard
    component, e.g. "/usr/lib/logdata-anomaly-miner/aminer/analysis/NewMatchPathDetector.py"
    as example.

  * Introduction of "AnalysisContext" to have common handle for
    all data required to perform the analysis. Therefore also
    the signature of "build_analysis_pipeline" in "config.py/analysis.py"
    has changed from

    def build_analysis_pipeline(aminer_config):

    to

    def build_analysis_pipeline(analysis_context):

    Old references to "aminer_config" within the configuration
    script have to be replaced by "analysis_context.aminer_config".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Thu, 21 Jul 2016 19:00:00 +0000