Fastcapa

Automated Installation

The best documentation is code that actually does this for you. An Ansible role that performs the entire installation procedure can be found at metron-deployment/roles/fastcapa. Use this to install Fastcapa or as a guide for manual installation. The automated installation assumes CentOS 7.1 and is directly tested against bento/centos-7.1.

Manual Installation

The following manual installation steps assume that they are executed on CentOS 7.1. Some minor differences may result if you use a different Linux distribution.

• Enable Transparent Huge Pages
• Install DPDK
• Install Librdkafka
• Install Fastcapa

Parameters

Fastcapa accepts three sets of parameters.

1. Command-line parameters passed directly to DPDK’s Environmental Abstraction Layer (EAL)
2. Command-line parameters that define how Fastcapa will interact with DPDK. These parametera are separated on the command line by a double-dash (--).
3. A configuration file that define how Fastcapa interacts with Librdkafka.

  -c COREMASK         Hexadecimal bitmask of cores to run on
  -l CORELIST         List of cores to run on
                      The argument format is <c1>[-c2][,c3[-c4],...]
                      where c1, c2, etc are core indexes between 0 and 128
  --lcores COREMAP    Map lcore set to physical cpu set
                      The argument format is
                            '<lcores[@cpus]>[<,lcores[@cpus]>...]'
                      lcores and cpus list are grouped by '(' and ')'
                      Within the group, '-' is used for range separator,
                      ',' is used for single number separator.
                      '( )' can be omitted for single element group,
                      '@' can be omitted if cpus and lcores have the same value                     

fastcapa -h

fastcapa -- -h

fastcapa [EAL options] -- [APP options]
  -p PORT_MASK        bitmask of ports to bind                     [0x01]
  -b RX_BURST_SIZE    burst size of receive worker                 [32]
  -w TX_BURST_SIZE    burst size of transmit worker                [256]
  -d NB_RX_DESC       num of descriptors for receive ring          [1024]
  -x TX_RING_SIZE     size of tx rings (must be a power of 2)      [2048]
  -q NB_RX_QUEUE      num of receive queues for each device        [1]
  -t KAFKA_TOPIC      name of the kafka topic                      [pcap]
  -c KAFKA_CONF       file containing configs for kafka client
  -s KAFKA_STATS      append kafka client stats to a file
  -h                  print this help message

[kafka-global]
metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092

Output

When running the probe some basic counters are output to stdout. Of course during normal operation these values will be much larger.

     ------ in ------  --- queued --- ----- out ----- ---- drops ----
[nic]               8               -               -               -
[rx]                8               0                8                0
[tx]                8               0                8                0
[kaf]               8               1                7                0

• [nic] + in: The ethernet device is reporting that it has seen 8 packets.
• [rx] + in: The receive workers have consumed 8 packets from the device.
• [rx] + out: The receive workers have enqueued 8 packets onto the transmission rings.
• [rx] + drops: If the transmission rings become full it will prevent the receive workers from enqueuing additional packets. The excess packets are dropped. This value will never decrease.
• [tx] + in: The transmission workers have consumed 8 packets.
• [tx] + out: The transmission workers have packaged 8 packets into Kafka messages.
• [tx] + drops: If the Kafka client library accepted fewer packets than expected. This value can increase or decrease over time as additional packets are acknowledged by the Kafka client library at a later point in time.
• [kaf] + in: The Kafka client library has received 8 packets.
• [kaf] + out: A total of 7 packets has successfully reached Kafka.
• [kaf] + queued: There is 1 packet within the rdkafka queue waiting to be sent.

Kerberos

The probe can be used in a Kerberized environment. Follow these additional steps to use Fastcapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.

• The Kafka broker is at kafka1:6667
• Zookeeper is at zookeeper1:2181
• The Kafka security protocol is SASL_PLAINTEXT
• The keytab used is located at /etc/security/keytabs/metron.headless.keytab
• The service principal is metron@EXAMPLE.COM

1. Build Librdkafka with SASL support (--enable-sasl).

   wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz  -O - | tar -xz
   cd librdkafka-0.9.4/
   ./configure --prefix=$RDK_PREFIX --enable-sasl
   make
   make install
   

2. Validate Librdkafka does indeed support SASL. Run the following command and ensure that sasl is returned as a built-in feature.

   $ examples/rdkafka_example -X builtin.features
   builtin.features = gzip,snappy,ssl,sasl,regex
   

   If it is not, ensure that you have libsasl or libsasl2 installed. On CentOS, this can be installed with the following command.

   yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
   

3. Grant access to your Kafka topic. In this example, it is simply named pcap.

   $KAFKA_HOME/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer \
     --authorizer-properties zookeeper.connect=zookeeper1:2181 \
     --add --allow-principal User:metron --topic pcap
   

4. Obtain a Kerberos ticket.

   kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
   

5. Add the following additional configuration values to your Fastcapa configuration file.

   security.protocol = SASL_PLAINTEXT
   sasl.kerberos.keytab = /etc/security/keytabs/metron.headless.keytab
   sasl.kerberos.principal = metron@EXAMPLE.COM
   

6. Now run Fastcapa as you normally would. It should have no problem landing packets in your kerberized Kafka broker.

Kafka Partitions

Parallelizing access to a topic in Kafka is achieved by defining multiple partitions. The greater the number of partitions, the more parallelizable access to that topic becomes. To achieve high throughput it is important to ensure that the Kafka topic in use has a large number of partitions, evenly distributed across each of the nodes in your Kafka cluster.

The specific number of partitions needed will differ for each environment, but at least 128 partitions has been shown to significantly increase performance in some environments.

Physical Layout

It is important to note the physical layout of the hardware when assigning worker cores to the probe. The worker cores should be on the same NUMA node or socket as the ethernet device itself. Assigning logical cores across NUMA boundaries can significantly impede performance.

The following commands can help identify logical cores that are located on the same NUMA node or socket as the ethernet device itself. These commands should be run when the device is still managed by the kernel itself; before binding the interface.

$ cat /sys/class/net/enp9s0f0/device/local_cpulist
0-7,16-23

The following command can be used to better understand the physical layout of the CPU in relation to NUMA nodes.

$ lscpu
...
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31

In this example enp9s0f0 is located on NUMA node 0 and is local to the logical cores 0-7 and 16-23. You should choose worker cores from this list.

CPU Isolation

Once you have chosen the logical cores to use that are local to the ethernet device, it also beneficial to isolate those cores so that the Linux kernel scheduler does not attempt to run tasks on them. This can be done using the isolcpus kernel boot parameter.

isolcpus=0,1,2,3,4,5,6,7

Device Limitations

Check the output of running the probe to ensure that there are no device limitations that you are not aware of. While you may have specified 16 receive queues on the command line, your device may not support that number. This is especially true for the number of receive queues and descriptors.

The following example shows the output when the number of receive descriptors requested is greater than what can be supported by the device. In many cases the probe will not terminate, but will choose the maximum allowable value and continue. This behavior is specific to the underlying device driver in use.

PMD: rte_enic_pmd: Rq 0 Scatter rx mode enabled
PMD: rte_enic_pmd: Rq 0 Scatter rx mode not being used
PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
PMD: rte_enic_pmd: Rq 1 Scatter rx mode enabled
PMD: rte_enic_pmd: Rq 1 Scatter rx mode not being used
PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
PMD: rte_enic_pmd: TX Queues - effective number of descs:32
PMD: rte_enic_pmd: vNIC resources used:  wq 1 rq 4 cq 3 intr 0

More Information

More information on this topic can be found in DPDK’s Getting Started Guide.

No free hugepages reported

Problem: When executing fastcapa it fails with the following error message.

EAL: No free hugepages reported in hugepages-1048576kB
PANIC in rte_eal_init():
Cannot get hugepage information

Solution: This can occur if any process that has been allocated THPs crashes and fails to return the resources.

• Delete the THP files that are not in use.

  rm -f /mnt/huge_1GB/rtemap_*
  

• If the first option does not work, re-mount the hugetlbfs file system.

  umount -a -t hugetlbfs
  mount -a
  

No ethernet ports detected

Problem: When executing fastcapa it fails with the following error message.

EAL: Error - exiting with code: 1
  Cause: No ethernet ports detected.

• Solution 1: The uio_pci_generic kernel module has not been loaded.

modprobe uio_pci_generic

• Solution 2: Ensure that the ethernet device is bound. Re-bind if necessary.

 dpdk-devbind --unbind "09:00.0"
 dpdk-devbind --bind=uio_pci_generic "09:00.0"
 dpdk-devbind --status