Removing Confluent Kafka from RHEL7 Confluent packages can simply be removed the same way they are installed. We can download the package and run the services separately. But if you have it installed through yum, it can be as easily removed. Here we are removing it using yum remove command. After getting a confirmation, it… Continue reading »
com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink While upgrading from Flink from 1.4 to a newer version 1.6.1, there are a few build issues. After we fix the issues, suddenly we started getting KryoException. Why are we getting this and how it this related to the upgrade? Let’s start with the exception message. If you try to… Continue reading »
Flink Upgrade SBT Assembly Merge Strategy We decided to upgrade Flink to 1.6.1. As we updated the code, it was running fine on the development machine. I realize that I started getting build issues as soon as I pushed out to source control. It was picked up by the Teamcity build agent and it threw… Continue reading »
JDK Tools – jps ( Java Process Tool ) Java Process Tool is an amazing JDK tool to check the details of java process on local or remote machines. It can be a very useful tool in the tool belt of a developer. By default, it shows the PID and short name of class. We… Continue reading »
Mocking Inner Singleton objects for Scala Tests using MockitoSugar This is a special case of testing when you need to test a type but dependencies come in all shapes and sizes. In order to test your class, you might not have any control on the dependencies. This is specially true when your dependencies are from… Continue reading »
Flink – Interactive Scala Shell Interactive scala shell is available in Flink download package. It allows us to run flink streams and batch processing statements in scala by building and executing the jobs directly created from the shell. We can simply call execute() method, which should request to run the job. The shell supports including… Continue reading »
Apache Flink – mapWithState on KeyedStream mapWithState operator provides a syntactic ease for keeping ValueState for map operations on KeyedStreams in Apache Flink. It is specially useful for doing a running count on the data. That is exactly what we are going to try in this post. Just remember that you don’t have to return… Continue reading »
Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »
KAFKA REST Proxy – Publishing Avro Messages to Kafka Apache Kafka supports message publishing through REST too. We can use Kafka REST proxy for this purpose. It supports http verbs including GET, POST and DELETE. Her is an example POST using curl which we will be trying to dissect throughout this post: Required Schema Apache… Continue reading »
Downloading Flink You can download flink from it’s Apache’s site. We are downloading Flink 1.6, which is the latest version available. https://flink.apache.org/downloads.html It requires Java version 8. Running Flink Cluster Let’s start the cluster. We can just run it directly from bin folder using start-cluster.sh utility available with the download. Alternatively we can run it… Continue reading »