Category Archives: Development

Kafka Connect – Externalizing Secrets – KIP 297

Kafka Connect – Externalizing Secrets – KIP 297 In order to connect with a data source or sink, we need to use credentials. Kafka Connect added support for specifying credential using config providers. The support for file config provider is available with the installation package. This is discussed in KIP 297. The KIP was released… Continue reading »

Securing JMX on Confluent Kafka

Securing JMX on Confluent Kafka Confluent kafka process start with these default arguments. You can see that JMX authentication is disabled by default. This is a security vulnerability and might lead to possible issues. Confluent kafka consists of the following services. We need to enable authentication for all of these services for JMX: Kafka Broker… Continue reading »

Kafka Tools – kafkacat – non-JVM Kafka producer / consumer

Kafka Tools – kafkacat – non-JVM Kafka producer / consumer

Kafka Tools – kafkacat – non-JVM Kafka producer / consumer kafkacat is an amazing kafka tool based on librdkafka library, which is a C/C++ library for kafka. It means that it doesn’t have dependency on JVM to work with kafka data as administrator. It can be used to consume and produce messages from kafka topics…. Continue reading »

Kafka Tools – kafka-delete-records

Kafka Tools – kafka-delete-records While working with kafka we, sometimes, need to purge records from a topic. This is specially needed in a development environment where we just want to get rid of some records and want to keep the other ones. Previously we have discussed how we can purge all records from a topic… Continue reading »

Confluent Hub Client

Confluent Hub Client Confluent Hub is an online repository for extensions and components for Kafka. Kafka is based on extensible model for many of its services. It allows plug-ins and extensions which makes it generic enough to be suitable for many real world streaming based applications. They include both Confluent and 3rd party components. Generally,… Continue reading »

Issues with Running Elasticseach from Command Line on Linux – bootstrap checks failed

Issues with Running Elasticseach from Command Line on Linux – bootstrap checks failed I ran into some issues running Elasticsearch on Linux. The issues and solutions might be of some use for others. vm.max_map_count [65530] is too low If you are getting the following error: Then you can resolve it by increasing the max_map_count to… Continue reading »