Category Archives: Uncategorized

Flink Upgrade SBT Assembly Merge Strategy

Flink Upgrade SBT Assembly Merge Strategy We decided to upgrade Flink to 1.6.1. As we updated the code, it was running fine on the development machine. I realize that I started getting build issues as soon as I pushed out to source control. It was picked up by the Teamcity build agent and it threw… Continue reading »

Connectors getting deleted after restart

Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:

Kafka CLI Tools – Add / Delete Topics

Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic

Kafka Streams – Interactive Queries (Intra Application Instance)

Kafka Streams – Interactive Queries (Intra Application Instance) Kafka Streams application need to maintain KTables in an internal application store. It maintains the store using RocksDB. It even allows to query this store during application lifecycle anywhere from the application parts. Let’s first create a simple application KafkaStreamsInteractiveQueriesApp. It is a simple Scala Application. In… Continue reading »

Setting up IntelliJ IDEA behind Proxy

Plugins Download First you need to set proxy for plugins download. Go to File\Settings and open Plugin tab. Get Proxy settings from internet explorer (Internet options) and update it here: org.scala-sbt#sbt;0.13.16: not found Here you need to use the same version build.propeties file as the one in sbt launcher. Check SBT Version on your machine… Continue reading »

SBT Behind Proxy

SBT Behind Proxy If you are running sbt behind firewall, you need to update proxy settings to be used: sbtconfig.txt Update the sbtconfig.txt where you have sbt installed: In addition to this, please make sure that sbt is added to $PATH variable. Using SBT from Git Bash Even though I had it configured as follows,… Continue reading »

Using TypeSafe’s Config In Scala

Using TypeSafe’s Config In Scala Our application behavior is generally defined with a set of configurations. Instead of hard-coding the values in the code, we like to define them separately in a configuration store. In this post, we are going to see how we can define configurations using TypeSafe’s config. Let’s create a SBT project…. Continue reading »