Kafka Connect – Single Message Transforms (SMTs)

Single Message Transforms were released with 0.10.2 release [ Release notes ]. It provides us the ability to transform a message before they get in or out of a connector using Kafka Connect.

Source Transforms

Source connector can be configured with a list of transforms. These transforms are applied operate on the data produced by a source connector. They work in a pipeline fashion where the output of one transform is provided as the input for the next transform. In the end, the data from the last transform is published to Kafka.

Source Transforms

Sink Transforms

Sink connectors can also be configured with a list of transforms. As consumer API pulls data from Kafka, it also goes through a list of transforms in a pipeline fashion. The data is then provided to the Sink connector, which can operate and push it to an actual sink e.g. database.

Sink Transforms

Available Transforms

Kafka Connect is installed with a number of connectors. Here is the list of available Transforms:

Available Transforms

Using Transforms for students topic

In the previous post, we saw how we saw how easily we can use off-the-shelf connectors to load data in or out of Kafka [http://www.alternatestack.com/development/app-development/confluent-using-off-the-shelf-connectors/]. As an example, we used FileStreamSourceConnector to load data from a file into a kafka topic. Here is how the message looked like in Kafka Topics UI:

Messages with empty key

Here we are using transform to pick timestamp from metadata and add as String field:

Good Reads

https://kafka.apache.org/documentation.html#connect_transforms
https://dzone.com/articles/real-time-data-pipelines-with-kafka-connect
https://www.slideshare.net/ConfluentInc/kafka-summit-nyc-2017-singe-message-transforms-are-not-the-transformations-youre-looking-for
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/
Kafka Summit Videos and Slides
https://jaxlondon.com/wp-content/uploads/2017/10/Real_Kafka_Implementations_of_the_Enterprise_Tim_Berglund.pdf
https://engineering.linkedin.com/blog/topic/kafka
https://github.com/airbnb/kafkat
Schema registry CLI
https://www.slideshare.net/ConfluentInc/kafka-summit-nyc-2017-singe-message-transforms-are-not-the-transformations-youre-looking-for
https://www.confluent.io/kafka-summit-nyc17/resource/