Kafka Tools – kafkacat – non-JVM Kafka producer / consumer

kafkacat is an amazing kafka tool based on librdkafka library, which is a C/C++ library for kafka. It means that it doesn’t have dependency on JVM to work with kafka data as administrator. It can be used to consume and produce messages from kafka topics. It also supports getting information about metadata information from kafka brokers and topics.

Getting kafkacat

The simplest way is to use docker. Confluent also has the image available on its repository. It’s also packaged and made available by others. You can find them here:

kafkacat-docker

I have a mac. It can also be installed using homebrew. But if you don’t have development tools installed, it might show some errors as follows:

kafkacat homebrew

It was clear what needs to be done here. You can just install developer tools using xcode-select –install as follows:

xcode-select –install

And it goes through the steps for installation successfully as follows:

xcode-select intall

xcode-select intall — progress

Now the installation runs successfully. You can see that it also has installed the dependencies automatically including librdkafka.

brew install kafkacat

Install jq

The outputs from the tool can also be requested in json format using -J switch. I would recommend you to install jq tool. The tool allows to pretty print the json on the console.

For Metadata

-L switch is used to get the metadata information. It can be used to get the metadata information for all topics in the cluster. You can also specify the particular topic you are interested in using the -t switch.

As Consumer

Most commonly, you might use kafkacat for consuming messages in an environment to see the details about the messages in a topic.

You can use the format expression with various parameters:

As Producer

In development environment specifically, one needs a tool to easily send messages on a topic to be consumed by a consumer service. kafkacat supports producing messages directly from the console.

It also supports loading the data from file and sending them to the topic. If we don’t use -l switch in this case, it would send the contents of whole file as one record.

Schema Registry Support

Apparently, there is still no schema registry support so please be careful with your messages in AVRO format.

schema-registry support