Confluent Hub Client Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. Generally,… Continue reading »
Removing Confluent Kafka from RHEL7 Confluent packages can simply be removed the same way they are installed. We can download the package and run the services separately. But if you have it installed through yum, it can be as easily removed. Here we are removing it using yum remove command. After getting a confirmation, it… Continue reading »
Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »
Custom Log4J Configuration file for Kafka In order to use a custom Log4J configuration file and to override the default one, we need to add the details in -Dlog4j.configuration. Here is how you can update the configuration path: As you can notice, we have also update the directory where logs are generated by overriding LOG_DIR.
Kafka Streams – Resetting Application State I the previous post, we discussed that we might need to reprocess data during development during application development. Since Kafka Streams ensures the application state, it doesn’t pull and reprocess data. In this case, you might find yourself keep waiting for the join operations to get triggered, but to… Continue reading »
Kafka Tools – kafka.tools.GetOffsetShell GetOffsetShell can be used to get the last offsets of a topic or individual partitions of a topic. The tool is available with your Kafka installation. It can be run using kafka-run-class. Here we are getting the offsets for Merged-SensorsHeartbeat topic.
Kafka Streams Config’s General Properties In this post, I just need to add Kafka Stream configuration which I have to use over and over again for a Kafka streams application. This can be used when developing a new Kafka Streams application. First we need to introduce configurations for Stream. They include details of broker and… Continue reading »
Kafka Streams: Converting KStream to KTable There are no methods in KStream to convert into a KTable. But there are workarounds to do that. They are as follows: Write on a Kafka Topic and read back as Kable We can also write to an intermediate Kafka Topic and read it back using StreamBuilder. Here is… Continue reading »
Using Landoop’s Schema Registry UI In this post, we are going to setup Schema Registry UI for Confluent’s schema registry using Docker Image for the tool. Schema Registry is an amazing tool by Landoop. It’s available in Github. Let’s first pull the image from Docker Hub. Now we can simply run the docker image by… Continue reading »
Installing Confluent Platform on Mac OS There are no direct download instructions available on the confluent platform yet to install the packages on Mac OS. I just want to share it with others just in case someone might need some help with this. Download the Archive The zip packages can be downloaded from here: Download…. Continue reading »