Connectors getting deleted after restart
Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:
Looking up ISIN / SEDOL / Other Identifiers
Looking up ISIN / SEDOL / Other Identifiers There is an amazing resource to look for details of a security based on their identifiers. http://isindb.com Currently it supports searching based on the following: ISIN CUSIP SEDOL
Kafka CLI Tools – Add / Delete Topics
Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic
Building Serde for Kafka Streams Application
Building Serde for Kafka Streams Application While developing Kafka Streams applications, I found myself requiring some utility code over and over. One such code is to build Serde for custom types. It is better if it is refactored into separate types and used when needed. It has two methods specificAvroSerde and genericAvroSerde. It must be… Continue reading »
Kafka Tools – kafka.tools.GetOffsetShell
Kafka Tools – kafka.tools.GetOffsetShell GetOffsetShell can be used to get the last offsets of a topic or individual partitions of a topic. The tool is available with your Kafka installation. It can be run using kafka-run-class. Here we are getting the offsets for Merged-SensorsHeartbeat topic.
Kafka Tools – Mirror Maker
Kafka Tools – Mirror Maker MirrorMaker is a Kafka tools for copying data from one cluster to the other. In this post we are going to see how we can run Mirror Maker to copy data from one cluster to the other. This can be specially useful when we want to copy data between two… Continue reading »
Serde and Kafka Streams
Serde and Kafka Streams In order to push / pull messages from Kafka we need to use Serde. I have found that if you create a Serializer / Deserialzer like following then it becomes really useful to create the Serde for your types. You can also use the method to create a GenericSerde to serialize… Continue reading »
Kafka Streams Config’s General Properties
Kafka Streams Config’s General Properties In this post, I just need to add Kafka Stream configuration which I have to use over and over again for a Kafka streams application. This can be used when developing a new Kafka Streams application. First we need to introduce configurations for Stream. They include details of broker and… Continue reading »
SBT 1.0 and IntelliJ IDEA Scala plugin – needs update
SBT 1.0 and IntelliJ IDEA Scala plugin – needs update I had to do a few updates to my Scala development tools, which hit me with a surprise when I tried to build one of Scala SBT applications. The build failed with misleading error messages. Digging deeper, I could find this message in one of… Continue reading »
Using Private Key for Authentication in teamcity
Using Private Key for Authentication in teamcity In this post, we are going to see how we can use ssh keys with teamcity. In order to use this, you need to have GitBash installed in your machine. It provides ssh-key tool to generate public/private keys. You need to to upload the private key file to… Continue reading »