Separate log file for connector The logs for Kafka connectors are generally added to connectDistributed.out. If you want to create a separate log file for your connector then you can update the following log4j.properties file. ../etc/kafka/connect-log4j.properties Add the following lines to the file: log4j.appender.connectorNameSpace=org.apache.log4j.RollingFileAppender log4j.appender.connectorNameSpace.DatePattern=’.’yyyy-MM-dd-HH log4j.appender.connectorNameSpace.MaxFileSize=10MB log4j.appender.connectorNameSpace.MaxBackupIndex=10 log4j.appender.connectorNameSpace.File=${kafka.logs.dir}/connect-my.log log4j.appender.connectorNameSpace.layout=org.apache.log4j.PatternLayout log4j.appender.connectorNameSpace.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L – %m%n… Continue reading »

Flink Upgrade SBT Assembly Merge Strategy We decided to upgrade Flink to 1.6.1. As we updated the code, it was running fine on the development machine. I realize that I started getting build issues as soon as I pushed out to source control. It was picked up by the Teamcity build agent and it threw… Continue reading »

Couchbase – Updating the Max Number of Buckets Per Cluster Couchbase sets a maximum limit of 10 buckets per cluster. Now this setting is configurable and REST API supports that. Here is how you can update it:

Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:

Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic