Kafka Data Collector
Contributors Download PDF of this topic
Cloud Insights uses this data collector to gather metrics from Kafka.
From Admin > Data Collectors, click +Data Collector. Under Services, choose Kafka.
Select the Operating System or Platform on which the Telegraf agent is installed.
If you haven’t already installed an Agent for collection, or you wish to install an Agent for a different Operating System or Platform, click Show Instructions to expand the Agent installation instructions.
Select the Agent Access Key for use with this data collector. You can add a new Agent Access Key by clicking the + Agent Access Key button. Best practice: Use a different Agent Access Key only when you want to group data collectors, for example, by OS/Platform.
Follow the configuration steps to configure the data collector. The instructions vary depending on the type of Operating System or Platform you are using to collect data.
The Kafka plugin is based on the telegraf’s Jolokia plugin. As such as a requirement to gather info from all Kafka brokers, JMX needs to be configured and exposed via Jolokia on all components.
Configuration was developed against Kafka version 0.11.0.2.
All the instructions below assume your install location for kafka is '/opt/kafka'. You can adapt instructions below to reflect your install location.
Jolokia Agent Jar
A version the Jolokia agent jar file must be downloaded. The version tested against was Jolokia agent 1.6.0.
Instructions below assume that the downloaded jar file (jolokia-jvm-1.6.0-agent.jar) is placed under the location '/opt/kafka/libs/'.
To configure Kafka Brokers to expose the Jolokia API, you can add the following in <KAFKA_HOME>/bin/kafka-server-start.sh, just before the 'kafka-run-class.sh' call:
export JMX_PORT=9999 export RMI_HOSTNAME=`hostname -I` export KAFKA_JMX_OPTS="-javaagent:/opt/kafka/libs/jolokia-jvm-1.6.0-agent.jar=port=8778,host=0.0.0.0 -Dcom.sun.management.jmxremote.password.file=/opt/kafka/config/jmxremote.password -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$RMI_HOSTNAME -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
Note that example above is using 'hostname -I' to setup the 'RMI_HOSTNAME' environment variable. In multiple IP machines, this will need to be tweaked to gather the IP you care about for RMI connections.
You can choose a different port for JMX (9999 above) and Jolokia (8778). If you have an internal IP to lock Jolokia onto you can replace the "catch all" 0.0.0.0 by your own IP. Notice this IP needs to be accessible from the telegraf plugin. You can use the option '-Dcom.sun.management.jmxremote.authenticate=false' if you don’t want to authenticate. Use at your own risk.
Objects and Counters
The following objects and their counters are collected:
Replica Manager Fetcher Max Lag
Additional information may be found from the Support page.