logdata-anomaly-miner (1.0.0-1) unstable; urgency=low * Ported code to Python 3 * Code cleanup using pylint * Added util/JsonUtil.py to encode byte strings for storing them as json objects * Added docs/development-procedures.txt which documents development procedures * New MissingMatchPathListValueDetector to detect stream interuption * Added parsing support for kernel IP layer martian package messages * Systemd parsing of apt invocation messages. Bugfixes: * AnalysisChild: handle remote control client connection errors correctly * Various bugfixes -- Markus Wurzenberger Tue, 2 Oct 2018 17:00:00 +0000 logdata-anomaly-miner (0.0.8-1) unstable; urgency=low Apart from bugfixes, new parsing and analysis components were added: * Base64StringModelElement * DecimalFloatValueModelElement * StringRegexMatchRule * EnhancedNewMatchPathValueComboDetector -- Roman Fiedler Tue, 30 May 2017 17:00:00 +0000 logdata-anomaly-miner (0.0.7-1) unstable; urgency=low The datetime parsing DateTimeModelElement was reimplemented to fix various shortcomings of strptime in Python and libc. This will require changes in configuration due to API changes, e.g.: -timeModel=DateTimeModelElement('time', '%b %d %H:%M:%S', 15, False) +timeModel=DateTimeModelElement('time', '%b %d %H:%M:%S') See /usr/lib/logdata-anomaly-miner/aminer/parsing/DateTimeModelElement.py source code documentation for currently supported datetime format options. The code for reading log input was improved to allow also input from UNIX sockets. Thus the configuration was changed to support those modes: -configProperties['LogFileList']=['/var/log/auth.log', ... +configProperties['LogResourceList'] = ['file:///var/log/auth.log', ... -- Roman Fiedler Mon, 9 Jan 2017 18:00:00 +0000 logdata-anomaly-miner (0.0.6-1) unstable; urgency=low The input IO-handling was redesigned, thus introducing following API changes. The changes are flaged with (D)eveloper and (U)ser to indicate if only developers of own AMiner addons are affected or also users may need to migrate their configuration. * Upper layers receive LogAtom objects instead of log lines, parsing data as separate parameters. Thus also separate paths for forwarding of parsed and unparsed atoms are not required any more. See below for details (D, U): * Update any own UnparsedAtomHandler/ParsedAtomHandlerInterface implementations to use new interface "input.AtomHandlerInterface" and access to additional information to new methods and fields (D): -from aminer.parsing import ParsedAtomHandlerInterface +from aminer.input import AtomHandlerInterface -class YourHandler(ParsedAtomHandlerInterface, ... +class YourHandler(AtomHandlerInterface, - def receiveParsedAtom(self, atomData, parserMatch): + def receiveAtom(self, logAtom): - timestamp=parserMatch.getDefaultTimestamp() + timestamp=logAtom.getTimestamp() + parserMatch=logAtom.parserMatch - print '%s' % atomData + print '%s' % logAtom.rawData * With parsed/unparsed atom processing path convergence, naming of other classes does not make sense any more (U): -from aminer.analysis import VolatileLogarithmicBackoffParsedAtomHistory +from aminer.util import VolatileLogarithmicBackoffAtomHistory - from aminer.analysis import ParsedAtomFilters + from aminer.analysis import AtomFilters - matchAction=Rules.ParsedAtomFilterMatchAction(... + matchAction=Rules.AtomFilterMatchAction(... - parsedAtomHandlers=[] - unparsedAtomHandlers=[] - analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory( - parsingModel, parsedAtomHandlers, unparsedAtomHandlers, ... + atomFilter=AtomFilters.SubhandlerFilter(None) + analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory( + parsingModel, [atomFilter], ... For handling of unparsed atoms: - unparsedAtomHandlers.append(SimpleUnparsedAtomHandler(anomalyEventHandlers)) + atomFilter.addHandler(SimpleUnparsedAtomHandler(anomalyEventHandlers), + stopWhenHandledFlag=True) For handling of parsed atoms: - parsedAtomHandlers.append(... + atomFilter.addHandler(... -- Roman Fiedler Fri, 4 Nov 2016 18:00:00 +0000 logdata-anomaly-miner (0.0.5-1) unstable; urgency=low Following API changes were introduced: * Lower input layers dealing with binary data stream reading, splitting into atoms and forwarding data to the parsing model were redesigned. Following configuration changes are required to adapt "config.py" and probably "analysis.py" to the new API: * analysisContext.registerComponent(): registerAsRawAtomHandler parameter not needed any more, can be removed. * SimpleParsingModelRawAtomHandler is not needed any more, that part can be replaced by configuration: # Now define the AtomizerFactory using the model. A simple line # based one is usually sufficient. from aminer.input import SimpleByteStreamLineAtomizerFactory analysisContext.atomizerFactory=SimpleByteStreamLineAtomizerFactory( parsingModel, parsedAtomHandlers, unparsedAtomHandlers, anomalyEventHandlers, defaultTimestampPath='/model/syslog/time') * SimpleUnparsedAtomHandler was moved from "aminer.events" to "aminer.input". -- Roman Fiedler Mon, 11 Oct 2016 18:00:00 +0000 logdata-anomaly-miner (0.0.4-1) unstable; urgency=low Following API changes were introduced: * Event handling (general): Change of EventHandlerInterface to include also eventSource as last parameter. See /usr/lib/logdata-anomaly-miner/aminer/events/__init__.py * VolatileLogarithmicBackoffEventHistory: Added event ID and source to stored tuple to allow unique identification of events. Split result of "getHistory()" to include "eventId, eventType, eventMessage, sortedLogLines, eventData, eventSource". -- Roman Fiedler Fri, 26 Aug 2016 15:15:00 +0000 logdata-anomaly-miner (0.0.3-1) unstable; urgency=low Following API changes were introduced: * To improve readability of configuration files, main parser, analysis and event classes were added to the submodule namespaces. After imports directly from the submodule, e.g. "from aminer.parsing import FixedDataModelElement", the name duplication "FixedDataModelElement.FixedDataModelElement" is not required any more, "FixedDataModelElement" is sufficient. Use "sed -i -e 's/Name.Name/Name/g' [files]" to adapt. * Component timing was restructured to allow forensic/realtime triggering. Therefore also clean interface was added, which is now also used to reduce redundant code in component registration. Old way: analysisContext.registerComponent(newMatchPathDetector, componentName=None, registerAsRawAtomHandler=False, registerAsTimeTriggeredHandler=True) New way: analysisContext.registerComponent(newMatchPathDetector, registerAsRawAtomHandler=False) For own custom time-triggered components, make sure to implement the "aminer.util.TimeTriggeredComponentInterface". Use any standard component, e.g. "/usr/lib/logdata-anomaly-miner/aminer/analysis/NewMatchPathDetector.py" as example. * Introduction of "AnalysisContext" to have common handle for all data required to perform the analysis. Therefore also the signature of "buildAnalysisPipeline" in "config.py/analysis.py" has changed from def buildAnalysisPipeline(aminerConfig): to def buildAnalysisPipeline(analysisContext): Old references to "aminerConfig" within the configuration script have to be replaced by "analysisContext.aminerConfig". -- Roman Fiedler Thu, 21 Jul 2016 19:00:00 +0000