Author Archives: Muhammad

Apache Flink – Collections & Streams

Apache Flink – Collections & Streams

While developing a streaming application, it is often necessary to use some inputs as collections. This can be specially useful for testing purposes. In this post we are trying to discuss how we can create a DataStream from a collection. After applying some operators on the stream, we are going to create an iterator from the elements of the stream.

StreamExecutionEnvironment supports creating streams from elements of a collection using fromElements method. Here we are using an integer collection. Here it returns DataStream[Int].

We are then using map operator to calculate double of the elements of each element. Finally DataStreamUtils is used to convert the stream to an iterator. We can now simply print out the elements of the iterator. Here is the output:

Output

Output

It is important that DataStreamUtils is used after executing the stream operators. Otherwise, it returns in the following error on executing the stream using StreamExecutionEnvironment.

Download Code

https://github.com/msiddiqi/ApacheFlingStreamsAndIterators

https://github.com/msiddiqi/ApacheFlingStreamsAndIterators

Apache Flink – Class Not found [org/apache/flink/streaming/api/scala/DataStream]

Apache Flink – Class Not found [org/apache/flink/streaming/api/scala/DataStream]
If you are developing a Flink app following examples on the web, you can most likely encounter the following error:

Generally, the problem is resolved by correctly setting the source folder in a scala project in IntelliJ but the weird thing is that it is happening for a dependency. Additionally, there are no compile issues. The actual problem is the dependency itself. Here are the recommended dependencies:

These dependencies expect a running flink. Just update the dependencies as follows:

And now you can see that the application is successfully running in IntelliJ and the breakpoints are being hit as expected:

flink_dependencies

flink_dependencies

Apache Flink – Change Port for web front end

Apache Flink – Change Port for web front end

Apache Flink runs the dashboard on port 8081. Since this is a common port there might be conflict with some other services running on the same machines. You might encounter this scenario especially during development when many services are running on the your development machine.

But the port can be changed. Just open conf/flink-conf.yaml file and update the rest.port property.

vi conf/flink-conf.yaml

vi conf/flink-conf.yaml

Now let’s start the cluster using bin/start-cluster.sh.

bin\start-cluster.sh

bin\start-cluster.sh

And flink is launched. You can verify by opening the dashboard in the browser. As you can see we are able top open the front-end using port 8081. The port is also correctly reflected in the job manager configuration details page as follows:

http://localhost:8089

http://localhost:8089

IntelliJ & Scala – Could not load main class

IntelliJ & Scala – Could not load main class

This is one of the most common error we see when we get new code from a source repository. We might also see it when we create a new Scala project. As we run the application, the console shows could not load main class

Could not load Main class

Could not load Main class

The solution is simple. Just open module setting and see if the source folder for code has been correctly specified for the module which is being used to run the application.

source folder in module

source folder in module

Just make sure that the same module is selected in your debug configuration in Use class path of module option.

classpath of module

classpath of module

IntelliJ – Add New Scala Class Option – Not Available

IntelliJ – Add New Scala Class Option – Not Available

You might get into this problem every time a new Scala project is created. By default, the support of Scala is not available. So when you right click your src folder to add Scala Class, you might be disappointed to see no option for adding a new Scala class is available.

Screen Shot 2018-09-01 at 5.00.33 PM

This can be resolved by following the following simple steps. Just right click your project and select Add Framework Support from the context menu shown.

Screen Shot 2018-09-01 at 5.00.00 PM

If you have Scala plugin installed, you should see the option for Scala available. Select the option and hit OK.

Screen Shot 2018-09-01 at 5.00.11 PM

You also might need to mark src folder as Source root.

Screen Shot 2018-09-01 at 5.04.47 PM

Now the option should be available to add a new Scala class.

Screen Shot 2018-09-01 at 5.05.00 PM

Angular Material Toggle Buttons Group with Binding

Angular Material Toggle Buttons Group with Binding

In this post we are going to create an Angular App to use Button toggle groups from Angular material. We are going to do data binding of these buttons and generate the buttons using a list in the code behind in the component class.

App

App

Let’s first create a simple Angular App using Angular CLI.

ng new button-toggle-app

ng new button-toggle-app

Now let’s install the necessary npm modules to use the toggle buttons.

npm install --save @angular/material @angular/cdk @angular/animations

npm install –save @angular/material @angular/cdk @angular/animations

Let’s import the material modules to app module. This allows them to be used in the whole module.

We need to update styles.css to use angular material theme:

Let’s introduce an array in the code behind (app.components.ts). We are going to use this array to create the list of toggle buttons in UI.

Here we are using the toggleButtons collection with *ngFor. We are binding the individual item to the value property of the toggle button. Please note that we are allowing multiple values to be selected in the button group. We are then displaying the selected options just below the button toggle group.

Default Options & Selected Values

Now let’s introduce the idea of default options when the page is loaded. We can bind value property in the button group to drive this. This can be bound with a string array in the code behind. Here is how we can change the template:

And here is how we can introduce the property in the code behind:

Handling Toggle Event

We can handle toggle events for individual button. We can even pass the selected / unselected value in the button group. It must be remembered that it handles the binding before emitting events so if you want to use the set value of the bound property, it should be alright.

Here is how we can update the code behind. We have just added a method to handle the value change of the buttons. We are just printing the current value and all the values currently selected by the button group.

Now we just need to use it in the template.

Code Repository

https://github.com/msiddiqi/angular-material-buttons-toggle

https://github.com/msiddiqi/angular-material-buttons-toggle

Kafka-Consumer-Groups – Kafka CLI Tools

Kafka-Consumer-Groups – Kafka CLI Tools

Kafka-Consumer-Groups is a CLI tool which can be used to get the message consumption from Kafka. The tool can be used to get the list of topics and partitions consumed by a consumer group. It also details if a consumer is lagging in consumption of the messages.

First we need to get the list of consumer groups. -list switch is used to get the list of consumer groups.

Now we have the details of a consumer group, we can get the detail of message consumption from the kafka topic and its partitions.

Here is the output:

Output Details

Output Details