Kafka Streams – Resetting Application State I the previous post, we discussed that we might need to reprocess data during development during application development. Since Kafka Streams ensures the application state, it doesn’t pull and reprocess data. In this case, you might find yourself keep waiting for the join operations to get triggered, but to… Continue reading »
Kafka Connect Issues: Found null value for non-optional schema In a custom connector, you might encounter the following issue: Here we have a schema defined for one of our custom connector: Now you would see the error when you create a SourceRecord with a value missing the non-optional fields.
Viewing systemctl logs for RHEL7 using journalctl The logs for systemctl service can be viewed using journalctl. Here we are viewing the logs for a process with PID 87920.
Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:
Looking up ISIN / SEDOL / Other Identifiers There is an amazing resource to look for details of a security based on their identifiers. http://isindb.com Currently it supports searching based on the following: ISIN CUSIP SEDOL
Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic
Building Serde for Kafka Streams Application While developing Kafka Streams applications, I found myself requiring some utility code over and over. One such code is to build Serde for custom types. It is better if it is refactored into separate types and used when needed. It has two methods specificAvroSerde and genericAvroSerde. It must be… Continue reading »
Kafka Tools – kafka.tools.GetOffsetShell GetOffsetShell can be used to get the last offsets of a topic or individual partitions of a topic. The tool is available with your Kafka installation. It can be run using kafka-run-class. Here we are getting the offsets for Merged-SensorsHeartbeat topic.
Kafka Tools – Mirror Maker MirrorMaker is a Kafka tools for copying data from one cluster to the other. In this post we are going to see how we can run Mirror Maker to copy data from one cluster to the other. This can be specially useful when we want to copy data between two… Continue reading »
Serde and Kafka Streams In order to push / pull messages from Kafka we need to use Serde. I have found that if you create a Serializer / Deserialzer like following then it becomes really useful to create the Serde for your types. You can also use the method to create a GenericSerde to serialize… Continue reading »