Kafka Tools – kafka-delete-records While working with kafka we, sometimes, need to purge records from a topic. This is specially needed in a development environment where we just want to get rid of some records and want to keep the other ones. Previously we have discussed how we can purge all records from a topic… Continue reading »
Apache Flink: History Server One of the problems we have faced running Apache Flink that we have a very limited windows for getting access to the details about failed / crashed jobs. The job details just disappear and cannot be retrieved especially form web front end. Here is how it looks like: Actually we are… Continue reading »
Kafka Connect – Single Message Transforms (SMTs) Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect. Source Transforms Source connector can be configured with a list of transforms. These transforms are… Continue reading »
KAFKA REST Proxy – Publishing Avro Messages to Kafka Apache Kafka supports message publishing through REST too. We can use Kafka REST proxy for this purpose. It supports http verbs including GET, POST and DELETE. Her is an example POST using curl which we will be trying to dissect throughout this post: Required Schema Apache… Continue reading »
Apache Flink – Collections & Streams While developing a streaming application, it is often necessary to use some inputs as collections. This can be specially useful for testing purposes. In this post we are trying to discuss how we can create a DataStream from a collection. After applying some operators on the stream, we are… Continue reading »
Neo4j – An Introduction A graph database allows data to be saved in terms of nodes and edges and their properties. This type of database suits the business cases where entities and their relationships defines the whole concept. They are faster to query too. In addition to storing data as nodes and relationships, they provide… Continue reading »