Category Archives: App Development

Confluent Hub Client

Confluent Hub Client Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. Generally,… Continue reading »

com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink

com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink While upgrading from Flink from 1.4 to a newer version 1.6.1, there are a few build issues. After we fix the issues, suddenly we started getting KryoException. Why are we getting this and how it this related to the upgrade? Let’s start with the exception message. If you try to… Continue reading »

Kafka Connect – Single Message Transforms (SMTs)

Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »

KAFKA REST Proxy – Publishing Avro Messages to Kafka

KAFKA REST Proxy – Publishing Avro Messages to Kafka Apache Kafka supports message publishing through REST too. We can use Kafka REST proxy for this purpose. It supports http verbs including GET, POST and DELETE. Her is an example POST using curl which we will be trying to dissect throughout this post: Required Schema Apache… Continue reading »

Apache Flink – Starting it up

Downloading Flink You can download flink from it’s Apache’s site. We are downloading Flink 1.6, which is the latest version available. https://flink.apache.org/downloads.html It requires Java version 8. Running Flink Cluster Let’s start the cluster. We can just run it directly from bin folder using start-cluster.sh utility available with the download. Alternatively we can run it… Continue reading »