JDK Tools – jps ( Java Process Tool ) Java Process Tool is an amazing JDK tool to check the details of java process on local or remote machines. It can be a very useful tool in the tool belt of a developer. By default, it shows the PID and short name of class. We… Continue reading »
Mocking Inner Singleton objects for Scala Tests using MockitoSugar This is a special case of testing when you need to test a type but dependencies come in all shapes and sizes. In order to test your class, you might not have any control on the dependencies. This is specially true when your dependencies are from… Continue reading »
Flink – Interactive Scala Shell Interactive scala shell is available in Flink download package. It allows us to run flink streams and batch processing statements in scala by building and executing the jobs directly created from the shell. We can simply call execute() method, which should request to run the job. The shell supports including… Continue reading »
Apache Flink – mapWithState on KeyedStream mapWithState operator provides a syntactic ease for keeping ValueState for map operations on KeyedStreams in Apache Flink. It is specially useful for doing a running count on the data. That is exactly what we are going to try in this post. Just remember that you don’t have to return… Continue reading »
Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »
KAFKA REST Proxy – Publishing Avro Messages to Kafka Apache Kafka supports message publishing through REST too. We can use Kafka REST proxy for this purpose. It supports http verbs including GET, POST and DELETE. Her is an example POST using curl which we will be trying to dissect throughout this post: Required Schema Apache… Continue reading »
Downloading Flink You can download flink from it’s Apache’s site. We are downloading Flink 1.6, which is the latest version available. https://flink.apache.org/downloads.html It requires Java version 8. Running Flink Cluster Let’s start the cluster. We can just run it directly from bin folder using start-cluster.sh utility available with the download. Alternatively we can run it… Continue reading »
Apache Flink – Collections & Streams While developing a streaming application, it is often necessary to use some inputs as collections. This can be specially useful for testing purposes. In this post we are trying to discuss how we can create a DataStream from a collection. After applying some operators on the stream, we are… Continue reading »
![Apache Flink – Class Not found [org/apache/flink/streaming/api/scala/DataStream]](http://www.alternatestack.com/wp-content/uploads/2018/09/Screen-Shot-2018-09-16-at-5.29.04-PM-200x140.png)
Apache Flink – Class Not found [org/apache/flink/streaming/api/scala/DataStream] If you are developing a Flink app following examples on the web, you can most likely encounter the following error: Generally, the problem is resolved by correctly setting the source folder in a scala project in IntelliJ but the weird thing is that it is happening for a… Continue reading »
Custom Log4J Configuration file for Kafka In order to use a custom Log4J configuration file and to override the default one, we need to add the details in -Dlog4j.configuration. Here is how you can update the configuration path: As you can notice, we have also update the directory where logs are generated by overriding LOG_DIR.