Category Archives: Development

Kafka Tools – kafka-delete-records

Kafka Tools – kafka-delete-records While working with kafka we, sometimes, need to purge records from a topic. This is specially needed in a development environment where we just want to get rid of some records and want to keep the other ones. Previously we have discussed how we can purge all records from a topic… Continue reading »

Confluent Hub Client

Confluent Hub Client Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. Generally,… Continue reading »

Issues with Running Elasticseach from Command Line on Linux – bootstrap checks failed

Issues with Running Elasticseach from Command Line on Linux – bootstrap checks failed I ran into some issues running Elasticsearch on Linux. The issues and solutions might be of some use for others. vm.max_map_count [65530] is too low If you are getting the following error: Then you can resolve it by increasing the max_map_count to… Continue reading »

com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink

com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink While upgrading from Flink from 1.4 to a newer version 1.6.1, there are a few build issues. After we fix the issues, suddenly we started getting KryoException. Why are we getting this and how it this related to the upgrade? Let’s start with the exception message. If you try to… Continue reading »

Kafka Connect – Single Message Transforms (SMTs)

Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »