Kafka Tools – kafka.tools.GetOffsetShell GetOffsetShell can be used to get the last offsets of a topic or individual partitions of a topic. The tool is available with your Kafka installation. It can be run using kafka-run-class. Here we are getting the offsets for Merged-SensorsHeartbeat topic.
Kafka Tools – Mirror Maker MirrorMaker is a Kafka tools for copying data from one cluster to the other. In this post we are going to see how we can run Mirror Maker to copy data from one cluster to the other. This can be specially useful when we want to copy data between two… Continue reading »
Serde and Kafka Streams In order to push / pull messages from Kafka we need to use Serde. I have found that if you create a Serializer / Deserialzer like following then it becomes really useful to create the Serde for your types. You can also use the method to create a GenericSerde to serialize… Continue reading »
Kafka Streams Config’s General Properties In this post, I just need to add Kafka Stream configuration which I have to use over and over again for a Kafka streams application. This can be used when developing a new Kafka Streams application. First we need to introduce configurations for Stream. They include details of broker and… Continue reading »
SBT 1.0 and IntelliJ IDEA Scala plugin – needs update I had to do a few updates to my Scala development tools, which hit me with a surprise when I tried to build one of Scala SBT applications. The build failed with misleading error messages. Digging deeper, I could find this message in one of… Continue reading »
Using Private Key for Authentication in teamcity In this post, we are going to see how we can use ssh keys with teamcity. In order to use this, you need to have GitBash installed in your machine. It provides ssh-key tool to generate public/private keys. You need to to upload the private key file to… Continue reading »
Change git password for windows in IntelliJ If you are using Windows Credentials for Git. Just open settings and search for password. Here you need to select “Do no save, forget passwords after restart”. You can restart your IntelliJ now. Now when you interact with Git again in a new IntelliJ session, it would ask… Continue reading »
Kafka Tools – kafka.tools.ReplayLogProducer Replay Log Producer is a Kafka tool available to copy messages from one Kafka Topic to the other in the same Broker cluster. This supports copying a limited or all messages from a Kafka Topic. The tool is specially useful in a development environment when we want to test something with… Continue reading »
Kafka Streams – Interactive Queries (Intra Application Instance) Kafka Streams application need to maintain KTables in an internal application store. It maintains the store using RocksDB. It even allows to query this store during application lifecycle anywhere from the application parts. Let’s first create a simple application KafkaStreamsInteractiveQueriesApp. It is a simple Scala Application. In… Continue reading »
Purging Data in a Kafka Topics If you have been working with Kafka for sometime, you might need to purge data in a particular topic. In a dev environment, you might worked it around by just publishing and consuming from a different topic. But there are some cases when you might want to purge data… Continue reading »