Author Archives: Muhammad

Running VSCode with venv

Running VSCode with venv

In order to use venv from within VSCode, you need to set the path of venv in Settings.

Setting venv path in settings

Setting venv path in settings

In order to run notebooks here, it needs to install the required packages:

install notebook package

install notebook package

Just go to Show and Run Commands.

Now you can see that your venv is available for selection.

Now you can connect to your jupyter notebook:

AWS Athena – Default Profile .aws – okta

AWS Athena – Default Profile .aws – okta

The query editor needs a connection and sometimes we want to use the credentials generated in .aws folder in the machine. They might have been generated by any process including Okta.

You need to override the following parameters with simba driver:

Credentials Provider
jdbc:awsathena://;AwsCredentialsProviderClass=com.simba.athena.amazonaws.auth.profile.ProfileCredentialsProvider;

Credentials Provider Argument
AwsCredentialsProviderArguments=default

You would be needing these additional arguments:

  1. Endpoint URL: athena..amazonaws.com
  2. OutputLocation
  3. Database
  4. S3Output Location

Using Azure Compute Instance in VSCode

Using Azure Compute Instance in VSCode

In order to develop machine learning flows through Azure, you first need to login to your Azure account. After installation of the required pre-requisite extensions, you should see option to login to your azure account.

Azure Account Login

Azure Account Login

It gives you an option to connect to your Azure account.

If you are successfully logged in, you should see the subscriptions:

Your compute instance should be available in your workspace:

Python – Anaconda Installation

Python – Anaconda Installation

Anaconda can be downloaded from the following link:

Anaconda Download

Anaconda Download

You can select the version of the installer suitable for your system. Here we are choosing the Mac version.

Mac Installer

Mac Installer

After downloading we can run it through the Terminal.

Anaconda Command Line

Anaconda Command Line

To get a list of available sub-commands run anaconda-h.

Anaconda Sub Commands

Anaconda Sub Commands

Separate log file for connector

Separate log file for connector

The logs for Kafka connectors are generally added to connectDistributed.out. If you want to create a separate log file for your connector then you can update the following log4j.properties file.

../etc/kafka/connect-log4j.properties

Add the following lines to the file:

log4j.appender.connectorNameSpace=org.apache.log4j.RollingFileAppender
log4j.appender.connectorNameSpace.DatePattern=’.’yyyy-MM-dd-HH
log4j.appender.connectorNameSpace.MaxFileSize=10MB
log4j.appender.connectorNameSpace.MaxBackupIndex=10
log4j.appender.connectorNameSpace.File=${kafka.logs.dir}/connect-my.log
log4j.appender.connectorNameSpace.layout=org.apache.log4j.PatternLayout
log4j.appender.connectorNameSpace.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L – %m%n

log4j.logger.kafka.connect.connectorNameSpace=INFO, connectorNameSpace
log4j.additivity.connector.name.space=false

Angular – Progressive Web Apps using Service Workers

Angular – Progressive Web Apps using Service Workers

Service workers enable creating progressive web applications. It is a script running in the web browser supporting caching of the application resources. They preserve the resources even after the user closes the tab and serves them when the application is requested again.

Setting up Angular Project

Let’s first create a sample angular app.

We can use ng serve to verify that the application has been created successfully.

We can open the application in the web browser and it does load successfully.

It must be remembered that ng serve does not work with service workers so we need to host the application externally through a web server. Let’s use http-server for this. It can simply be installed through npm.

Just make sure that you are able to run it successfully:

Now let’s run our sample PWA app using http-server. Here -c-1 would disable caching.

Adding Service Worker Support

Let’s first verify that our application couldn’t be served offline. We can set the browser to appear as offline through Chrome Dev Tools.

Now we can add support for service worker by adding @angular/pwa.

As we build the application using ng build –prod, we can see that the browser is able to load the application offline using Service Worker.

We can also see Network tab to see service worker in action:

go – segmentation fault: 11

go – segmentation fault: 11

Running go after a few days, I notice that it is crashing with segmentation 11 error. In this post, we will try to get rid of this error in order to run go on our machine. Actually, I need this as a dependency to run some other development tool.

Reading through the internet, it seems that the program is attempting to access a memory that it shouldn’t. Let’s see if running with sudo would be of any help. It doesn’t…

As microfocus explains, it can also be caused by mismatched binaries. This is possible that some new development tools have messed my dev environment up. Let’s first remove go and then reinstall it.

Now we can download the latest go version from here. Just to note that we are running this on macOS Catalina:

Let’s download and run through the installation:

Now you can notice that we are successfully able to run go.

Hyperledger Fabric – Terminology

Hyperledger Fabric – Terminology

Hyperledger Fabric is private and permissioned. In order to gain access, the peers need to be enrolled through a Membership Service Provider (MSP) . It has a deterministic consensus algorithm.

Asset

An asset holds a state and has ownership. They are key / value pairs representing a value, which enables to exchange anything with monetary value. They can be tangible or intangible. They can be modified using chaincode transactions on a channel ledger i.e. the state changes of assets are recorded as transaction on the ledger.

Chaincode

Chaincode is a software that defines the asset. It also defines the transaction instruction to modify the asset. The businesses sign off on the business logic. Chaincode allow these business to interact with the ledger.

Membership Service Provider

They are sometimes also referred as Membership Identity Services. They provides IDs to the authenticated participants which can be used to define permission to the ledger.

Access Control List (ACL)

ACL is used for defining authorization on the ledger and chaincodes. They use IDs provided by MSP.

Node Types

There are two node types. They are Peer nodes and ordering nodes. Peer nodes can batch execute and verify transaction. These transactions are ordered and propagated through ordering nodes. These node types provide efficiency and scalability. A consensus protocol is defined for ordering transactions.

Ordering nodes also maintain the list of organizations that are allowed to create channels. This list of organizations is called consortium. This list is kept in the “order system channel”. This list is administered by an orderer node.Orderer also enforce access control on channels i.e. the updates to the configuration of these channels. After update the configuration block is propogated to peers.

Fabric’s Ledger

Fabric’s ledger is based on both blockchain log and current state of database. The current state can be easily queried. The log is maintained to keep asset provenance in order to provide asset creation and state changes by various members.

There is one ledger per channel and each peer maintains a copy of ledger of the channel they are a member of.

Channels

Hyperledger uses channel to provide privacy to the ledger. Only the participants of the channel have visibility to the assets and transactions on the channel. This is unlike other blockchains where all participants have access to the public ledger. So channels are restrictive message paths defining the subset of participants.

Ordering Service Implementations

In the latest version of Hyperledger fabric, there are three implementations of ordering services including Solo, Kafka and Raft.

Multitenant Fabric

A fabric can have multiple consortiums(list of organizations allowed to create channels) rendering the blockchain as multi-tenant.