Confluent – Using off-the-shelf-connectors with Landoop’s Kafka Connect UI

In this post we are going to see how easy it is to run a Kafka Connector in order to source data in Kafka using a well designed tool like Landoop’s Kafka Connect UI.

We can use Kafka Connect’s REST API to determine what connectors are available for our cluster. Here we are pulling this using /connector-plugins resource [http://localhost:8083/connector-plugins].

The same list is available in Kafka-connect-ui (discussed in an earlier post: Kafka-connect-ui). Here we are searching for file connectors. This is included as one of the default connectors available with the simple installation.

filestream-source-connector

Let’s provide the necessary configuration for FileStreamSourceConnector. This is used to pull data from a file into a kafka topic. Here we are pulling the contents of /tmp/students.csv file into students topic. As you can notice, we can easily pause, restart or delete the connector here.

connector-running

Here is how Landoop’s Kafka Connect UI Dashboard looks like:

Kafka Connect UI – Dashboard

We can also verify using Kafka Connect REST API. Here we are hitting the /connectors resource; which lists all the connectors currently running.

/connectors

In order to verify that the connector has actually loaded the file’s contents, we can use Kafka-topics-ui. We are simply pulling the docker image for the landoop’s tool and running it as follows:

Here we can select students topics and verify the data being pulled into this topic.

Connect Source data in Kafka Topics UI

Let’s update the contents of the file by adding some more contents. And we can see the items are correctly reflected in the topics.

topic-updated