Parser Chaining

Aggregating many different types sensors into a single data source (e.g. syslog) and ingesting that aggregate sensor into Metron is a common pattern. It is not obvious precisely how to manage these types of aggregate sensors as they require two-pass parsing. This document will walk through an example of supporting this kind of multi-pass ingest.

Multi-pass parser involves the following requirements:

• The enveloping parser (e.g. the aggregation format such as syslog or plain CSV) may contain metadata which should be ingested along with the data.
• The enveloping sensor contains many different sensor types

High Level Solution

At a high level, we continue to maintain the architectural invariant of a 1-1 relationship between logical sensors and storm topologies. Eventually this relationship may become more complex, but at the moment the approach is to construct a routing parser which will have two responsibilities:

• Parse the envelope (e.g. syslog data) and extract any metadata fields from the envelope to pass along
• Route the unfolded data to the appropriate kafka topic associated with the enveloped sensor data

Because the data emitted from the routing parser is just like any data emitted from any other parser, in that it is a JSON blob like any data emitted from any parser, we will need to adjust the downstream parsers to extract the enveloped data from the JSON blob and treat it as the data to parse.

Aggregated Parsers with Parser Chaining

Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor.

Say, there were three sensors (bro, snort and yaf). Instead of creating a topology per sensor, all 3 can be run in a single aggregated parser. It is also possible to aggregate a subset of these parsers (e.g. run bro as it’s own topology, and aggregate the other 2).

The step to start an aggregated parsers then becomes

$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s bro,snort,yaf

which will result in a single storm topology named bro__snort__yaf to run.

Aggregated parsers can be specified using the Ambari Metron config as well under Services -> Metron -> Configs -> ‘Parsers’ tab -> ‘Metron Parsers’ field. The grouping is configured by enclosing the desired parsers in double quotes.

Some examples of specifying aggregated parsers are as follows:

• “bro,snort,yaf” –> Will start a single topology named bro__snort__yaf
• “ciscopixA,ciscopixB”,yaf,“squid,ciscopixC” –> Will start three topologies viz. ciscopixA__ciscopixB, yaf and squid__ciscopixC

Architecting a Parser Chaining Solution in Metron

Currently the approach to fulfill this requirement involves a couple knobs in the Parser infrastructure for Metron.

Consider the case, for instance, where we have many different TYPES of messages wrapped inside of syslog. As an architectural abstraction, we would want to have the following properties:

• separate the concerns of parsing the individual types of messages from each other
• separate the concerns of parsing the individual types of messages from parsing the envelope

{
  "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
  ,"sensorTopic" : "my_topic"
  ,"rawMessageStrategy" : "ENVELOPE"
  ,"rawMessageStrategyConfig" : {
      "messageField" : "payload",
      "metadataPrefix" : ""
  }
  , "parserConfig": {
     "columns" : { "f1": 0,
                 , "f2": 1,
                 , "f3": 2
                 } 
   }
}

{
  "meta_f1" : "val1",
  "payload" : "foo,bar,grok",
  "original_string" : "2019 Jul, 01: val1 foo,bar,grok",
  "timestamp" : 10000
}

{
  "meta_f1" : "val1",
  "f1" : "foo",
  "f2" : "bar",
  "f3" : "grok",
  "original_string" : "2019 Jul, 01: val1 foo,bar,grok",
  "timestamp" : 10002
}