Category Archives: Uncategorized

Separate log file for connector

Separate log file for connector The logs for Kafka connectors are generally added to connectDistributed.out. If you want to create a separate log file for your connector then you can update the following log4j.properties file. ../etc/kafka/connect-log4j.properties Add the following lines to the file: log4j.appender.connectorNameSpace=org.apache.log4j.RollingFileAppender log4j.appender.connectorNameSpace.DatePattern=’.’yyyy-MM-dd-HH log4j.appender.connectorNameSpace.MaxFileSize=10MB log4j.appender.connectorNameSpace.MaxBackupIndex=10 log4j.appender.connectorNameSpace.File=${kafka.logs.dir}/connect-my.log log4j.appender.connectorNameSpace.layout=org.apache.log4j.PatternLayout log4j.appender.connectorNameSpace.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L – %m%n… Continue reading »

Flink Upgrade SBT Assembly Merge Strategy

Flink Upgrade SBT Assembly Merge Strategy We decided to upgrade Flink to 1.6.1. As we updated the code, it was running fine on the development machine. I realize that I started getting build issues as soon as I pushed out to source control. It was picked up by the Teamcity build agent and it threw… Continue reading »

Connectors getting deleted after restart

Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:

Kafka CLI Tools – Add / Delete Topics

Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic

Kafka Streams – Interactive Queries (Intra Application Instance)

Kafka Streams – Interactive Queries (Intra Application Instance) Kafka Streams application need to maintain KTables in an internal application store. It maintains the store using RocksDB. It even allows to query this store during application lifecycle anywhere from the application parts. Let’s first create a simple application KafkaStreamsInteractiveQueriesApp. It is a simple Scala Application. In… Continue reading »

Setting up IntelliJ IDEA behind Proxy

Plugins Download First you need to set proxy for plugins download. Go to File\Settings and open Plugin tab. Get Proxy settings from internet explorer (Internet options) and update it here: org.scala-sbt#sbt;0.13.16: not found Here you need to use the same version build.propeties file as the one in sbt launcher. Check SBT Version on your machine… Continue reading »

SBT Behind Proxy

SBT Behind Proxy If you are running sbt behind firewall, you need to update proxy settings to be used: sbtconfig.txt Update the sbtconfig.txt where you have sbt installed: In addition to this, please make sure that sbt is added to $PATH variable. Using SBT from Git Bash Even though I had it configured as follows,… Continue reading »