Couchbase – Updating the Max Number of Buckets Per Cluster Couchbase sets a maximum limit of 10 buckets per cluster. Now this setting is configurable and REST API supports that. Here is how you can update it:
Deleting Connectors from Kafka Connect A connector can be deleting using Kafka Connect’s REST API. Here is the curl command:
Kafka Streams – Resetting Application State I the previous post, we discussed that we might need to reprocess data during development during application development. Since Kafka Streams ensures the application state, it doesn’t pull and reprocess data. In this case, you might find yourself keep waiting for the join operations to get triggered, but to… Continue reading »
Kafka Connect Issues: Found null value for non-optional schema In a custom connector, you might encounter the following issue: Here we have a schema defined for one of our custom connector: Now you would see the error when you create a SourceRecord with a value missing the non-optional fields.
Viewing systemctl logs for RHEL7 using journalctl The logs for systemctl service can be viewed using journalctl. Here we are viewing the logs for a process with PID 87920.
Connectors getting deleted after restart Kafka connectors get deleted when you restart connector process after a few days. In order to resolve that we need to enable log compaction on the following topics:
Looking up ISIN / SEDOL / Other Identifiers There is an amazing resource to look for details of a security based on their identifiers. http://isindb.com Currently it supports searching based on the following: ISIN CUSIP SEDOL
Kafka CLI Tools – Add / Delete Topics Deleting Topic /bin/kafka-topics –delete –zookeeper localhost:2181 –topic MyTopic Creating Topic /bin/kafka-topics –create –zookeeper localhost:2181 –partitions 8 –replication-factor 3 –topic MyTopic
Building Serde for Kafka Streams Application While developing Kafka Streams applications, I found myself requiring some utility code over and over. One such code is to build Serde for custom types. It is better if it is refactored into separate types and used when needed. It has two methods specificAvroSerde and genericAvroSerde. It must be… Continue reading »
Kafka Tools – kafka.tools.GetOffsetShell GetOffsetShell can be used to get the last offsets of a topic or individual partitions of a topic. The tool is available with your Kafka installation. It can be run using kafka-run-class. Here we are getting the offsets for Merged-SensorsHeartbeat topic.