In order to use a custom Log4J configuration file and to override the default one, we need to add the details in -Dlog4j.configuration. Here is how you can update the configuration path:
As you can notice, we have also update the directory where logs are generated by overriding LOG_DIR.
Apache Flink runs the dashboard on port 8081. Since this is a common port there might be conflict with some other services running on the same machines. You might encounter this scenario especially during development when many services are running on the your development machine.
But the port can be changed. Just open conf/flink-conf.yaml file and update the rest.port property.
vi conf/flink-conf.yaml
Now let’s start the cluster using bin/start-cluster.sh.
bin\start-cluster.sh
And flink is launched. You can verify by opening the dashboard in the browser. As you can see we are able top open the front-end using port 8081. The port is also correctly reflected in the job manager configuration details page as follows:
http://localhost:8089
This is one of the most common error we see when we get new code from a source repository. We might also see it when we create a new Scala project. As we run the application, the console shows could not load main class
Could not load Main class
The solution is simple. Just open module setting and see if the source folder for code has been correctly specified for the module which is being used to run the application.
source folder in module
Just make sure that the same module is selected in your debug configuration in Use class path of module option.
classpath of module
You might get into this problem every time a new Scala project is created. By default, the support of Scala is not available. So when you right click your src folder to add Scala Class, you might be disappointed to see no option for adding a new Scala class is available.
This can be resolved by following the following simple steps. Just right click your project and select Add Framework Support from the context menu shown.
If you have Scala plugin installed, you should see the option for Scala available. Select the option and hit OK.
You also might need to mark src folder as Source root.
Now the option should be available to add a new Scala class.
In this post we are going to create an Angular App to use Button toggle groups from Angular material. We are going to do data binding of these buttons and generate the buttons using a list in the code behind in the component class.
App
Let’s first create a simple Angular App using Angular CLI.
ng new button-toggle-app
Now let’s install the necessary npm modules to use the toggle buttons.
npm install –save @angular/material @angular/cdk @angular/animations
Let’s import the material modules to app module. This allows them to be used in the whole module.
We need to update styles.css to use angular material theme:
Let’s introduce an array in the code behind (app.components.ts). We are going to use this array to create the list of toggle buttons in UI.
Here we are using the toggleButtons collection with *ngFor. We are binding the individual item to the value property of the toggle button. Please note that we are allowing multiple values to be selected in the button group. We are then displaying the selected options just below the button toggle group.
Now let’s introduce the idea of default options when the page is loaded. We can bind value property in the button group to drive this. This can be bound with a string array in the code behind. Here is how we can change the template:
And here is how we can introduce the property in the code behind:
We can handle toggle events for individual button. We can even pass the selected / unselected value in the button group. It must be remembered that it handles the binding before emitting events so if you want to use the set value of the bound property, it should be alright.
Here is how we can update the code behind. We have just added a method to handle the value change of the buttons. We are just printing the current value and all the values currently selected by the button group.
Now we just need to use it in the template.
https://github.com/msiddiqi/angular-material-buttons-toggle
Kafka-Consumer-Groups is a CLI tool which can be used to get the message consumption from Kafka. The tool can be used to get the list of topics and partitions consumed by a consumer group. It also details if a consumer is lagging in consumption of the messages.
First we need to get the list of consumer groups. -list switch is used to get the list of consumer groups.
Now we have the details of a consumer group, we can get the detail of message consumption from the kafka topic and its partitions.
Here is the output:
Output Details
Couchbase sets a maximum limit of 10 buckets per cluster. Now this setting is configurable and REST API supports that. Here is how you can update it:
A connector can be deleting using Kafka Connect’s REST API. Here is the curl command:
I the previous post, we discussed that we might need to reprocess data during development during application development. Since Kafka Streams ensures the application state, it doesn’t pull and reprocess data. In this case, you might find yourself keep waiting for the join operations to get triggered, but to your disappointment, you might not see the breakpoints being hit.
There are two alternate ways which can be used during application development and debugging ensuring the reprocessing of data.
Here is the streams configuration for our Kafka Streams application:
One possible solution is to use the new application state each time you need to re-process in your development environment. You can just use a random number for the configuration. It is very common to destroy the whole confluent environment using confluent destroy command. So if it looks non-manageable anytime, you can always destroy it. I wouldn’t recommend this as there is an alternate solution available which is also very easy to use.
Alternatively you can use Kafka Application reset tool. Here we are resetting the state of application with myApplicationId id. After executing the command, the offset should be set to the beginning of input topics. It would also delete all intermediate topics.
https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html
In a custom connector, you might encounter the following issue:
Here we have a schema defined for one of our custom connector:
Now you would see the error when you create a SourceRecord with a value missing the non-optional fields.