Kafka Tools – kafka-delete-records While working with kafka we, sometimes, need to purge records from a topic. This is specially needed in a development environment where we just want to get rid of some records and want to keep the other ones. Previously we have discussed how we can purge all records from a topic… Continue reading »
Apache Flink: History Server One of the problems we have faced running Apache Flink that we have a very limited windows for getting access to the details about failed / crashed jobs. The job details just disappear and cannot be retrieved especially form web front end. Here is how it looks like: Actually we are… Continue reading »
Confluent Hub Client Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. Generally,… Continue reading »
Issues with Running Elasticseach from Command Line on Linux – bootstrap checks failed I ran into some issues running Elasticsearch on Linux. The issues and solutions might be of some use for others. vm.max_map_count [65530] is too low If you are getting the following error: Then you can resolve it by increasing the max_map_count to… Continue reading »
Removing Confluent Kafka from RHEL7 Confluent packages can simply be removed the same way they are installed. We can download the package and run the services separately. But if you have it installed through yum, it can be as easily removed. Here we are removing it using yum remove command. After getting a confirmation, it… Continue reading »
com.esotericsoftware.kryo.KryoException: Unusual solution upgrading Flink While upgrading from Flink from 1.4 to a newer version 1.6.1, there are a few build issues. After we fix the issues, suddenly we started getting KryoException. Why are we getting this and how it this related to the upgrade? Let’s start with the exception message. If you try to… Continue reading »
Flink Upgrade SBT Assembly Merge Strategy We decided to upgrade Flink to 1.6.1. As we updated the code, it was running fine on the development machine. I realize that I started getting build issues as soon as I pushed out to source control. It was picked up by the Teamcity build agent and it threw… Continue reading »
JDK Tools – jps ( Java Process Tool ) Java Process Tool is an amazing JDK tool to check the details of java process on local or remote machines. It can be a very useful tool in the tool belt of a developer. By default, it shows the PID and short name of class. We… Continue reading »
Mocking Inner Singleton objects for Scala Tests using MockitoSugar This is a special case of testing when you need to test a type but dependencies come in all shapes and sizes. In order to test your class, you might not have any control on the dependencies. This is specially true when your dependencies are from… Continue reading »
Flink – Interactive Scala Shell Interactive scala shell is available in Flink download package. It allows us to run flink streams and batch processing statements in scala by building and executing the jobs directly created from the shell. We can simply call execute() method, which should request to run the job. The shell supports including… Continue reading »