kafkactl - Kafka Management Tool

kafkactl - CLI for Apache Kafka, Zookeeper and Burrow Management.

Current Features:

• Search / Filter for Groups and Topics
• Show Topic Partition Details - Offsets, Replicas, Leaders, etc.
• Show Group Details - Lag, Offsets, Members, etc.
• Display / Modify Topic Configs
• Create / Delete Topics
• Delete Consumer Groups
• Reset Partition Offsets for a Group
• Tail Topics in realtime
• Produce to a Specific Topic Partition or all Partitions at once
• Perform a Preferred Replica Election for a target topic or for all topics
• Increase / Decrease Replicas
• Migrate Topics between Brokers
• Query burrow details
• Monitor burrow lag via Terminal Graph
• Explore Zookeeper Paths
• Create and Delete Zookeeper Values
• Pass stdin to create Kafka messages or Zookeeper Values

ToDo:

• Add Security Features
• Add Metrics Testing
• Test Avro Serialization

kafkactl is actively developed with new features being added and tested. Thus, ongoing optimization and re-factoring will occur so ensure you are aware of the latest releases.

kafkactl tool - Get Started

From Source

• Requires Go Version 1.11+

git clone https://github.com/jbvmio/kafkactl
cd kafkactl/
go build -o $GOPATH/bin/kafkactl

# If inside your go path:
GO111MODULE=on go build -o $GOPATH/bin/kafkactl

Homebrew

# First time install:
brew tap jbvmio/tap
brew install jbvmio/tap/kafkactl

# Upgrading
brew upgrade jbvmio/tap/kafkactl

Docker

docker pull docker.io/jbvmio/kafkactl:latest
docker run --rm jbvmio/kafkactl -B broker:9092 get topics --lag
docker run --rm jbvmio/kafkactl -Z zookeeper:2181 ls /brokers/ids/2
docker run --rm jbvmio/kafkactl config --sample

Manual Download and Install (Linux, Windows, Mac)

• Download the latest kafkactl tool appropriate for your platform, extract and install to a valid directory, preferably located somewhere in your configured $PATH.
• Run "kafkactl config --sample" to generate and display a sample config. Create your config and place as $HOME/.kafkactl.yaml

Alternatively, pass all arguments using flags instead of using a config file when running kafkactl:

# kafkactl --broker brokerhost01:9092 admin create topic newtopic01 --partitions 15 --rfactor 3

kafkactl - Example Commands

example config commands

kafkactl config --sample
kafkactl config view
kafkactl config get-contexts
kafkactl config current-context
kafkactl config use cluster1

example config file (~/.kafkactl.yaml)

contexts:
  prod-atl01:
    name: prod-atl01
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
  prod-atl02:
    name: prod-atl02
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
  prod-atl03:
    name: prod-atl03
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
current-context: prod-atl01
config-version: 1

Usage

kafkactl -h

kafkactl: Kafka Management Tool

Usage:
  kafkactl [flags]
  kafkactl [command]

Examples:
  kafkactl --context <contextName> get brokers

Available Commands:
  admin       Kafka Admin Actions
  burrow      Show Burrow Lag Evaluations <wip>
  config      Show and Edit kafkactl config
  describe    Get Kafka Details
  get         Get Kafka Information
  help        Help about any command
  logs        Get Messages from a Kafka Topic
  send        Send/Produce Messages to a Kafka Topic
  version     Print kafkactl version and exit
  zk          Zookeeper Actions

Flags:
  -B, --broker string      Specify a single broker target host:port - Overrides config.
      --burrow string      Specify a single burrow endpoint http://host:port - Overrides config.
      --cfg string         config file (default is $HOME/.kafkactl.yaml)
  -C, --context string     Specify a context.
  -x, --exact              Find exact matches.
  -h, --help               help for kafkactl
  -o, --out string         Change Output Format - yaml|json.
  -v, --verbose            Display additional info or errors.
      --version string     Specify a client version.
  -Z, --zookeeper string   Specify a single zookeeper target host:port - Overrides config.

kafkactl admin -h

Available Commands:
  create      Create Kafka Resources
  delete      Delete Kafka Resources
  get         Get Kafka Configurations
  move        Move Partitions using Stdin
  set         Set Kafka Configurations

kafkactl zk -h

Zookeeper Actions

Usage:
  kafkactl zk [flags]
  kafkactl zk [command]

Available Commands:
  create      Create Zookeeper Paths and Values
  delete      Delete Zookeeper Paths and Values
  ls          Print Zookeeper Paths and Values

Flags:
  -h, --help         help for zk
  -o, --out string   Change Output Format - yaml|json.

Global Flags:
  -B, --broker string      Specify a single broker target host:port - Overrides config.
      --burrow string      Specify a single burrow endpoint http://host:port - Overrides config.
      --cfg string         config file (default is $HOME/.kafkactl.yaml)
  -C, --context string     Specify a context.
  -x, --exact              Find exact matches.
  -v, --verbose            Display additional info or errors.
      --version string     Specify a client version.
  -Z, --zookeeper string   Specify a single zookeeper target host:port - Overrides config.

Example output:

kafka

# kafkactl

Brokers:  5
 Topics:  208
 Groups:  126

Cluster: (Kafka: v1.1)
  brokerhost01:9092/1
* brokerhost02:9092/2
  brokerhost03:9092/3
  brokerhost04:9092/4
  brokerhost05:9092/5

(*)Controller

# kafkactl get group mygroup05

GROUPTYPE  GROUP      COORDINATOR
consumer   mygroup05  broker03:9092/3

kafkactl get group mygroup05 --lag

GROUP      TOPIC      PART  MEMBER                    OFFSET  LAG
mygroup05  mytopic05  14    mygroup05-77885078-77abc  97      0
mygroup05  mytopic05  1     mygroup05-77885078-77abc  76      0
mygroup05  mytopic05  2     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  9     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  12    mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  3     mygroup05-77885078-77abc  135     0
mygroup05  mytopic05  8     mygroup05-77885078-77abc  90      0
mygroup05  mytopic05  7     mygroup05-77885078-77abc  83      0
mygroup05  mytopic05  4     mygroup05-77885078-77abc  111     0
mygroup05  mytopic05  6     mygroup05-77885078-77abc  114     0
mygroup05  mytopic05  0     mygroup05-77885078-77abc  2329    0
mygroup05  mytopic05  5     mygroup05-77885078-77abc  85      0
mygroup05  mytopic05  11    mygroup05-77885078-77abc  139     0
mygroup05  mytopic05  10    mygroup05-77885078-77abc  125     0
mygroup05  mytopic05  13    mygroup05-77885078-77abc  97      0

zookeeper

# kafkactl zk ls /cluster/id

VALUE: /cluster/id
 {"version":"1","id":"wwI7vYwOShKnEqVzDhhUYqi"}

# kafkactl zk ls /

PATH: /
 cluster
 controller
 brokers
 zookeeper
 admin
 isr_change_notification
 log_dir_event_notification
 controller_epoch
 consumers
 burrow
 latest_producer_id_block
 config

# kafkactl zk --create /testpath
Successfully Created: /testpath

# echo 'my testvalue from stdin' | kafkactl zk --create /testpath/testvalue
Successfully Created: /testpath/testvalue

# kafkactl zk ls /testpath/testvalue

VALUE: /testpath/testvalue
 my testvalue from stdin

burrow

kafkactl burrow mygroup07 -x

BURROW      GROUP      TOPIC      PARTITION  LAG  TOPICLAG  STATUS
dc01-kafka  mygroup07  mytopic07  0          0    0         OK
dc01-kafka  mygroup07  mytopic07  1          0    0         OK
dc01-kafka  mygroup07  mytopic07  2          0    0         OK
dc01-kafka  mygroup07  mytopic07  3          0    0         OK
dc01-kafka  mygroup07  mytopic07  4          0    0         OK
dc01-kafka  mygroup07  mytopic07  5          0    0         OK
dc01-kafka  mygroup07  mytopic07  6          0    0         OK
dc01-kafka  mygroup07  mytopic07  7          0    0         OK
dc01-kafka  mygroup07  mytopic07  8          0    0         OK
dc01-kafka  mygroup07  mytopic07  9          0    0         OK
dc01-kafka  mygroup07  mytopic07  10         0    0         OK
dc01-kafka  mygroup07  mytopic07  11         0    0         OK
dc01-kafka  mygroup07  mytopic07  12         0    0         OK
dc01-kafka  mygroup07  mytopic07  13         0    0         OK
dc01-kafka  mygroup07  mytopic07  14         0    0         OK