Confluent Components & Their HTTP Ports
It is very important to remember the ports of individual components of Confluent Platform. The following is the list of default ports for the respective components. It must be remembered that these ports can be overridden during deployment.
Confluent components Ports
More details can be found on this page on confluent.io where this is copied from:
Kafka and Confluent logs in Log File
During development and debugging, it is very useful to see traces for Kafka and confluent in log file to determine the issues. The traces are specially useful when we use DSL for Kafka Streams. Here is the logback configuration to use logback appenders for kafka and confluent logs.
Log4J Dependency with Kafka
Many of the Kafka artifacts have Log4J dependencies. If you are using logback then your logging stops all of a sudden. Here is an example for one of our sample app. Here we have added such dependency and the App seems to have issues with multiple SL4J bindings. It has finally picked up Log4J instead.
If we look at this log, clearly the other binding is coming from Log4J. If we look at the Project tab, we can also see log4J artifact being added here.
Here is our current build.sbt. It has “io.confluent” % “kafka-avro-serializer” % “1.0”, which is pulling up log4J as a dependency.
The solution is simple. We just need to tell SBT not to pull Log4J as dependency of this library.
And now we can see that the artifact for Log4J is removed from the external libraries.
log4j dependency exclusion
Packaging configurations outside Jars with SBT Native Plugin
In this post we are going to see how we can separately package any files (especially config and log configuration) out of packaged jar when we use sbt native packager for packaging our scala application. Here we are building from our last post.
In order to use sbt native packager; first we need to add the plugin in our packages configuration file (packages.sbt) under project folder.
Now we need to enable the plugin. Here JavaAppPackaging would package up jar files for our application. We can also notice a couple of mappings configurations. They copy application.dev.conf and logback.dev.xml files from resources folder to conf folder in the resulting package.
We have very simple configurations for our application.conf and application.dev.conf. We are simply keeping config.environment configuration. It has been assigned Debug and Dev values for debug and dev environments.
In order to demonstrate what configuration file is being used; here we are just pulling up config.environment configuration and logging it. Here we are using typesafe configs for handling with configuration files. If we simply run this code in IntelliJ debugger, you can notice that it pulls up application.conf file by default, hence writing Debug message to the logs.
Let’s simply package it using sbt. Here we are running sbt universal:packageBin to package it.
And this is how it packages it. You can notice that it has copied application.dev.conf and logback.dev.xml separately in conf folder.
Let’s simple run it using. We can notice that it has picked up application.conf file for configuration and has written Debug to the logs.
We can override configuration file using -Dconfig.file argument. We can also override logback configuration file using -Dlogback.configurationFile argument.
But below we have just overridden the configuration file. You can notice that it has written Dev to the logs. This shows that it has correctly picked up application.dev.conf file as specified in the startup argument.
avro-tools in Scala Projects – Don’t…
In this post we are going to discuss how Avro-tools dependency can mess up your logging in a Scala project. Let’s create a sample Scala SBT project.
We update the build.sbt as follows:
Here are we adding dependencies for logback in addition to avro-tools dependency.
Let’s create a simple Main and add logging here.
Your editor (IntelliJ) might show you that there are multiple sources for LoggerFactory. Both jar files has the same package too. This should ring some alarms. You must remember that avro-tools is not adding org.sl4j as a dependency jar but it provides an implementation of LoggerFactory embedded into its own jar file. So you cannot use exclusion here.
Let’s also add configuration for logback to specify details of our required appenders.
But there is no log file created. Now just remove avro-tools dependency from your sbt file and now you can see the log file created. So, definitely avro-tools dependency has messed it up.
But you might have avro types added to your project. You wouldn’t be able to compile now as the required types are missing. But you don’t need the avro-tools here for this purpose. You can simply add dependency for “org.apache.avro” % “avro” % “1.8.2”, which is the latest version as of this blog post.
This should have no effect on your logging.