Apache Metron on Amazon EC2

Prerequisites

The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.

• Ansible 2.4.0+
• Python 2.7.11
• Maven 3.3.9

Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent Homebrew project.

1. Install Homebrew by running the following command in a terminal. Refer to the Homebrew home page for the latest installation instructions.

   /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
   

2. With Homebrew installed, run the following command in a terminal to install all of the required tools and dependencies.

   brew update
   brew tap caskroom/versions
   brew cask install java8 vagrant virtualbox
   brew install maven git node
   

3. Install Ansible by following the instructions here.

4. Ensure that a public SSH key is located at ~/.ssh/id_rsa.pub.

   $ cat ~/.ssh/id_rsa.pub
   ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost
   

   If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.

   ssh-keygen -t rsa
   

5. Ensure the JAVA_HOME environment variable is set

   export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home"
   

   Notice: You must replace the path with the installed JDK version path

Amazon Web Services

If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.

1. Head over to Amazon Web Services and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.

2. Create a set of user credentials through Amazon’s Identity and Access Management (IAM) dashboard. On the IAM dashboard menu click “Users” and then “Create New User”. Provide a name and ensure that “Generate an access key for each user” remains checked. Download the credentials and keep them for later use.

3. While still in Amazon’s Identity and Access Management (IAM) dashboard, click on the user that was previously created. Click the “Permissions” tab and then the “Attach Policy” button. Attach the following policies to the user.

   • AmazonEC2FullAccess
   • AmazonVPCFullAccess

4. Apache Metron uses the official, open source CentOS 6 Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon’s terms and conditions. Navigate to the web page for this AMI and click the “Continue” button. Choose the “Manual Launch” tab then click the “Accept Software Terms” button.

Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us.

Deploy Metron

1. Use the Amazon access key by exporting its values via the shell’s environment. This allows Ansible to authenticate with Amazon EC2. For example:

   export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
   export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
   

   Notice: You must replace the access key values above with values from your own access key.

2. Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default.

   $ ./run.sh
   Metron Environment [metron-test]: my-metron-env
   ...
   

   The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.

Explore Metron

1. After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.

   TASK [debug] *******************************************************************
   ok: [localhost] => {
       "Success": [
           "Apache Metron deployed successfully",
           "   Metron  @  http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
           "   Ambari  @  http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
           "   Sensors @  ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
           "For additional information, see https://metron.apache.org/'"
       ]
   }
   

2. Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user centos will not require a password as it will authenticate with the pre-defined SSH key.

   ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
   

Multiple Environments

This process can support provisioning of multiple, isolated environments. Simply change the env settings in conf/defaults.yml. For example, you might provision separate development, test, and production environments.

env: metron-test

Selective Provisioning

To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command:

./run.sh --tags="ec2,sensors"

Setting REST API Profile

By default, EC2 is deployed with the dev Spring profile. This simply instructs the REST API to automatically setup default test users [user, user1, user2, admin] with password “password” as opposed to requiring the user to manually create users in the REST database. You can change this default functionality by removing the metron_spring_profiles_active setting in metron-deployment/roles/ambari_config/vars/small_cluster.yml. You can view more detail on the REST Spring profiles here.

Custom SSH Key

By default, the playbook will attempt to register your public SSH key ~/.ssh/id_rsa.pub with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the key_file property in conf/defaults.yml.

For example, generate a new SSH key for Metron that will be stored at ~/.ssh/my-metron-key.

$ ssh-keygen -q -f ~/.ssh/my-metron-key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Add the path to the newly created SSH public key to conf/defaults.yml.

key_file: ~/.ssh/metron-private-key.pub

Error: [unsupported_operation_exception] custom format isn’t supported

This error might be seen within Metron’s default dashboard in Kibana 4. This occurs when the index templates do not exist for the Snort, Bro or YAF indices in Elasticsearch.

The dashboard expects fields to be of a certain type. If the index templates have not been loaded correctly, the data types for the fields in these indices will be incorrect and the dashboard will display this error.

Error: ‘No handler was ready to authenticate…Check your credentials’

TASK [Define keypair] **********************************************************
failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
  r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
  sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
  "msg": "No handler was ready to authenticate. 1 handlers were checked.
  ['HmacAuthV4Handler'] Check your credentials"}

$ echo $AWS_ACCESS_KEY_ID
AKIAI6NRFEO27E5FFELQ

$ echo $AWS_SECRET_ACCESS_KEY
vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed

Error: ‘OptInRequired: … you need to accept terms and subscribe’

TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
product you need to accept terms and subscribe. To do so please visit
http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
to retry, use: --limit @playbook.retry

Error: ‘PendingVerification: Your account is currently being verified’

TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => PendingVerification: Your account is currently
being verified. Verification normally takes less than 2 hours. Until your
account is verified, you may not be able to launch additional instances or
create additional volumes. If you are still receiving this message after more
than 2 hours, please let us know by writing to aws-verification@amazon.com. We
appreciate your patience."}
to retry, use: --limit @playbook.retry

Error: ‘Instance creation failed => InstanceLimitExceeded’

TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => InstanceLimitExceeded: You have requested more
instances (11) than your current instance limit of 10 allows for the specified
instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
request an adjustment to this limit."}
to retry, use: --limit @playbook.retry

Error: ‘SSH encountered an unknown error during the connection’

TASK [setup] *******************************************************************
fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
  "changed": false, "msg": "SSH encountered an unknown error during the
  connection. We recommend you re-run the command using -vvvv, which will enable
  SSH debugging output to help diagnose the issue", "unreachable": true}