Enrichment

Introduction

The enrichment module is a module dedicated to taking the data from the parsers that have been normalized into the Metron data format (e.g. a JSON Map structure with original_message and timestamp) and

• Enriching messages with external data from data stores (e.g. hbase) by adding new fields based on existing fields in the messages.
• Marking messages as threats based on data in external data stores
• Marking threat alerts with a numeric triage level based on a set of Stellar rules.

Deployment Options

There is currently one option for running enrichments in Metron, which is as a Storm topology.

Submodules

• metron-enrichment-common - this module houses the prepackaged enrichment configuration by sensor. It also contains the core enrichment and threat intelligence processing functionality.
• metron-common-storm - this module is home to Storm-specific code such as Flux files and Storm Bolts.

Enrichments List

Metron provides an HBase table for storing enrichments. The rowkeys are a combination of a salt (for managing RegionServer hotspotting), indicator (this would be the search value, e.g. “192.168.1.1”), and type (whois, geoip, etc.).

This approach performs well for both inserts and lookups, but poses a challenge when looking to get an up-to-date list of all the current enrichments. This is of particular concern for the Management UI where it’s desirable to provide an accurate list of all available enrichment types. A table scan is undesirable because it results in a performance hit for inserts and reads. The alternative approach that mitigates these performance bottlenecks is to leverage a custom HBase Coprocessor which will listen to postPut calls from the RegionServer, extract the enrichment type from the rowkey, and perform an insert into a separate enrichment_list HBase table.

See more about configuring the coprocessor here Enrichment Coprocessor