Serde and Kafka Streams In order to push / pull messages from Kafka we need to use Serde. I have found that if you create a Serializer / Deserialzer like following then it becomes really useful to create the Serde for your types. You can also use the method to create a GenericSerde to serialize… Continue reading »
Kafka Streams Config’s General Properties In this post, I just need to add Kafka Stream configuration which I have to use over and over again for a Kafka streams application. This can be used when developing a new Kafka Streams application. First we need to introduce configurations for Stream. They include details of broker and… Continue reading »
SBT 1.0 and IntelliJ IDEA Scala plugin – needs update I had to do a few updates to my Scala development tools, which hit me with a surprise when I tried to build one of Scala SBT applications. The build failed with misleading error messages. Digging deeper, I could find this message in one of… Continue reading »
Using Private Key for Authentication in teamcity In this post, we are going to see how we can use ssh keys with teamcity. In order to use this, you need to have GitBash installed in your machine. It provides ssh-key tool to generate public/private keys. You need to to upload the private key file to… Continue reading »
Change git password for windows in IntelliJ If you are using Windows Credentials for Git. Just open settings and search for password. Here you need to select “Do no save, forget passwords after restart”. You can restart your IntelliJ now. Now when you interact with Git again in a new IntelliJ session, it would ask… Continue reading »
Kafka Tools – kafka.tools.ReplayLogProducer Replay Log Producer is a Kafka tool available to copy messages from one Kafka Topic to the other in the same Broker cluster. This supports copying a limited or all messages from a Kafka Topic. The tool is specially useful in a development environment when we want to test something with… Continue reading »
Kafka Streams – Interactive Queries (Intra Application Instance) Kafka Streams application need to maintain KTables in an internal application store. It maintains the store using RocksDB. It even allows to query this store during application lifecycle anywhere from the application parts. Let’s first create a simple application KafkaStreamsInteractiveQueriesApp. It is a simple Scala Application. In… Continue reading »
Purging Data in a Kafka Topics If you have been working with Kafka for sometime, you might need to purge data in a particular topic. In a dev environment, you might worked it around by just publishing and consuming from a different topic. But there are some cases when you might want to purge data… Continue reading »
Kafka Streams: Converting KStream to KTable There are no methods in KStream to convert into a KTable. But there are workarounds to do that. They are as follows: Write on a Kafka Topic and read back as Kable We can also write to an intermediate Kafka Topic and read it back using StreamBuilder. Here is… Continue reading »
Kafka Streams – Merging multiple KStream (s) In this post we are going to see how we can merge messages from multiple topics into a single topic in using Kafka Streams. In the example scenario, we will be using heart beat data for four sensors and merge into a single topic as follows: Here we… Continue reading »