Monthly Archives: December 2014

Running Kafka on Windows

Apache has provided an amazing documentation for Kafka. It also includes step-by-step guide to run download and run kafka. Unfortunately, the documentation misses any info for running it on windows platform. I really had to struggle running it on my windows box, so i thought i should add it here for anyone else going through the same issues.

Step I [Download Kafka Binaries]

First of all, we can download kafka from the following:

The download is a tar gzip compressed file with tgz extension. So first we need to un-compress it before using any files. There are a number of tools available for this. I have 7-zip installed on my machine, which also supports extracting contents from this type of compressed file. Here I have downloaded 0.8.11 with the scala version 2.10.

Just extract the contents of the archive into a folder.

Step II – Install support for running shell (*.sh) scripts

Now the second step of the kafka documentation suggests running Zookeeper directly. For a windows machine there is no built-in support for this, so you need to add tools for running the shell script. If you already have Git client installed, there is already a support, otherwise, install Cygwin making sure that you are selecting utils for installation.

Step III – Updating the compatible batch files

But if you just run it now, it just throws the following error message. Here I am using Cygwin terminal.

Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

I have seen a lot of people have this issue. Apparently, it is because of the batch files provided with the download. It is in Windows sub-directory of kafka folder.

I am using 64-bit Windows 8.1 machine. The solution that worked for me is to replace the win batch files with the ones provided here.

Step – IV Handling spaces in JAVA_HOME Environment Variable

I have JRE installed on this windows machine. The environment variable, JAVA_HOME, is set to the installation directory which has spaces in its path.

Running with this path set would introduce another problem.

If you clearly look at the error message, it is apparent that, somehow, kafka-run-class.sh script is being executed, which is throwing this error on line 153. Let’s do the universal solution for handling spaces in file paths, enclose it in quotes.

Step V – Missing Server JVM components

Now If you just have JRE installed, as I had on this machine, you should see the error message for missing server JVM.

In order to just get over this problem, you can just create a new folder, and name it server. Now copy the contents of client folder to this new folder.

Step VI – Unrecognized VM Option UseCompressedOops

Now as we move forward and run it, we are faced with another error. Only this time, it doesn’t tell where is the actual problem. This is the error message.

Unrecognized VM option ‘UseCompressedOops’

Just removing this JVM option, would let you get rid of this error message.

Step VII – QuorumPeerMain – Once Again

After going through all the above steps, reality hits you and you end up with the same error message that you started with. And that makes it really funny. Has all this effort been fruitless. Actually, No!!!

Since we are running it in windows, we are having class path problem. Since we already have cygwin installed, we can use cygpath. Just update line 153 in kafka-run-class again and use cygpath with class path.

Finally!!!

Finally, after going through all this hassle, you can have zookeeper running on your machine. It looks so pretty 🙂

Now you can continue with the next instructions for running kafka server, creating topics, sending message from producer and picking them up from consumer.

Zindabad!!!

Kafka – Introduction and Terminologies

Kafka – Introduction & Terminologies
Kafka is an apache project originally introduced by LinkedIn as an open source project. It is a distributed, fast, durable and scalable publish / subscribe messaging system. This messaging is also persistent with constant time performance [ O(1) ] even with many terabytes of storage. It also supports parallel data load into hadoop.

Conceptual Kafka

From the 40,000 feet high view, there are three entities in the kafka world. There are producers (publish message). These messages are published to kafka cluster. The messages are then pulled by consumers.

Broker

As we just discussed, Kafka is a distributed message platform. It runs a cluster with one or more servers. Each server in the kafka cluster is called Broker.

Topics & Partitions

Kafka messages are published to a topic. They can be referred as categories of messages. The producers publish their message on a topic, and consumers subscribe to topics for message consumption. Each topic can have one or more partitions, which are immutable, ordered sequence of messages. These partitions can be on the same broker, or they can span across multiple brokers.

@SiddiqiMuhammad yes, that is exactly what the partitions are for 🙂

— Apache Kafka (@apachekafka) December 22, 2014

Producer needs to decide which particular partition it needs to publish message to. It may do so in a round robin fashion. It can also be done based on some semantic partition function (key based).

Kafka introduced fault tolerance by introducing leaders and followers. The messages on a partition can be replicated on different servers. In this case, one of the servers would be called as Leader, and the rest are called followers. Messages are always published to the leader. They are then passively replicated by all the followers. In case of leader failure, one of the follower takes over as a leader without causing any effect to the consumers.

Mapping between Partitions & Consumers

Kafka keeps the ordering by one-to-one mapping between partition and consumer in a consumer group. This is similar to basic Domain & Range concept for Functions (single valued) in basic calculus. It can be an injective or non-injective surjective function. Each partition is assigned to only one consumer in a consumer group. More than one partitions can be assigned to one consumer. There can be no consumer without an assigned partition. The below example is definitely like a non-injective surjective function.

Consumer Pull Vs Partition Push

Kafka is based on pull mechanism. A consumer pulls messages from the assigned partition when it is ready.

Handling Ordering of Messages

Kafka guarantees ordering of messages on partition level. So if a topic has more than one partitions, the order of messages on different partitions cannot be guaranteed. It looks like, this could be even worse if those partitions are selected on a round robin fashion.

How would more than one producer use round robin fashion for publishing messages as it seems for round robin, it should be a central point of publishing???

Queuing Vs Publish / Subscribe

In a queuing system, a message can only be consumed by a single consumer. In a publish / subscribe based mechanism, it can be consumed by a number of consumers. Kafka can be configured as either of them. It is a queuing system for each consumer group. So if you have one consumer group, it is a queuing system.

https://cwiki.apache.org/confluence/display/KAFKA/Index

Beacon Configuration – Gimbal

Beacon Configuration
After getting the beacons delivered, beacons must be configured with appropriate settings based on the required specification. We cannot directly apply a setting to a Gimbal Beacon but we need to follow certain steps for this purpose. I thought this is a good idea to list them here.

Activating Gimbal Beacons

When you receive a Gimbal beacon, you need to first activate it. This would require beacon’s factory Id. For a Series 10 beacon, you need to open the beacon casing to find it. Now you can add it in Gimbal Manager online application:

Configuring Gimbal Beacons

Configuring a gimbal beacon is a three step process. This makes use of Gimbal Manager online application and Gimbal manager iOS app. A configuration can be applied to all beacons added to your inventory in Gimbal Manager online application.

Step – I [ Defining Configuration]

Gimbal provides an online platform to manage beacon configurations. It also gives you the ability to keep track of configurations applied to any particular beacon. The configuration is defined using Gimbal Manager. The configuration is based on the selected beacon type.

There are two possible beacon types. They are iBeacon and Gimbal beacon types. Here Gimbal type is a simple enough configuration. Here we can configure if we want to detect beacon only when the app is on the screen (Foreground only) or when it is not on user screen (Foreground or Background).

It must be remembered that iBeacon specification is only supported for a firmware version 1.6.1 and above.

There are two types of antennas in a beacon device. They are Omnidirectional and Directional types.

The measured power can be defined with a value in the range of 0 to -110 dBm.

The transmission power can be defined with a value in the range of 0 to -23 dBm.

Step – II [ Assigning Configuration ]

After creating the configuration, we can assign to a beacon. Since we need to add all beacons to Gimbal Manager application. This is just a matter of selecting a configuration from a drop down listing all your configurations defined in the previous step.

Step – III [ Applying Configuration on a Beacon ]

Now the configurations can be applied on a beacon using Beacon configuration manager app. This is an iPhone app, available for free.

I know this is disappointing, but there is no android app available yet. Let’s hope for it and keep asking…

@SiddiqiMuhammad Thanks for your interest in Gimbal. We currently offer a free Gimbal Beacon Manager app for iOS via iTunes.

— Gimbal (@Gimbal) December 15, 2014

BLE Beacons – Modes of Operation

Beacons are Bluetooth smart devices which transmit signals at regular intervals. These signals can be scanned by an devices. These devices can pass on the info of beacon of interest to an app, which uses it to determine proximity. In order to be recognized, beacons must be configured with certain information. For iBeacon case, they are Proximity UUID, major and minor numbers. So, in order to provide these features, a beacon can support a number of modes. In this mode, we will be trying to see different modes supported by some major bacon providers.

All beacons have a Beacon mode for their basic operation. In order to update settings, they have a configuration mode. Actually Bluetooth reception consumes power, since beacons don’t have to be accepting any reception signals, they are more efficient with power consumption. Some beacons are powered with a number of features and sensors, they support different modes for providing those features.

Estimote

There are two modes of operations for Estimote beacons. They are as follows:

Advertisement mode: This is the default mode. In this mode, beacon advertises for UUID, Major and minor numbers. These advertisements can be used by an app to determine required proximity. This is a transmit only mode.
Connectivity mode: In connectivity mode, Estimote’s beacon allows connection with the apps built using Estimote’s SDK. This mode can be used to collect info from beacon. The info includes sensor’s data (accelerometer and temperature), battery level, OS version, hardware revision, Tx power etc. This mode is also used for firmware update. This mode requires authentication for your app.

Estimote’s Power Modes

In the middle of 2014 (June), estimote released a number of updates to its iBeacons. One such update was introduction of power modes, which allow us to save battery upto 50%. Estimote introduced to non-mutually exclusive modes. They are Basic Battery Saving & Smart Battery Saving modes. Estimote claims that these modes can be used to save upto 30% and 50% power respectively. Here Smart Battery Savings looks at the pattern for the users around iBeacon and adjusts power consumption accordingly using adaptive algorithms. [Further Details]

I had my doubts if this would work if we don’t use Estimote API / SDK for our app. So I asked, and apparently, it does.

@SiddiqiMuhammad @Estimote Sure, they will, as those are at hardware settings.

— Ola Puchta (@Esti_Ola) December 10, 2014

Motorola’s MPact Beacon Tags

There are three modes of operations for MPact beacons. They are as follows:

iBeacon mode: This mode is similar as estimote’s advertisement mode. In this mode, MPact beacon just advertises for UUID, major and minor numbers at regular intervals.
Battery Save mode: In this mode, tag’s unique identifier and mac address are transmitted. Motorola claims that battery life doubles when this mode is used. But user must launch app when entering the location in order to determine location.
MPact mode: This is hybrid of above two modes. This provides info of battery level, which makes fleet management easier. Additionally, it doesn’t require an app to be running in the foreground in order to determine location.

These modes affects the maximum battery life for MPact beacon tags. This is from MPact spec.

Gelo

There are two modes of operations. They are as follows:

Beacon mode: This is a simple mode when beacon advertises for UUID, major and minor numbers.
Configuration mode: This mode is an exclusive mode. As soon as a beacon enters in this mode, all advertisements are disabled. Beacon enters in this mode by swiping a magnet across its top. Now Gelo app is used to connect and configure it. After configuration is done, the beacon is restarted in Advertisement mode. [Further details on GitHub]

iBKS

iBKS beacons have the following modes of operations:

Reception mode: iBKS beacon can also work in reception mode. But we need to select this option during configuration by providing password. So this is not turned-on by default, resulting in power savings.
Configuration mode Like other beacons, iBKS’s configuration mode allows us to configure advertisement packet’s details. Entering in configuration mode is quite similar to Gelo Beacon. It also requires a magnet to be swiped on it to enter configuration mode. The Hall sensor detects magnetic field and device enters in configuration mode.

Gimbal Beacons

Gimbal beacon can run in two different modes. For gimbal, it is called Beacon Type. These types are mutually exclusive. It must be remembered that all beacon configurations are created / updated using Gimbal Manager. These configuration also include ‘Beacon Type‘. We can then apply the configuration on a beacon.

These modes are as follows:

iBeacon Mode: This mode is based on apple’s iBeacon specification. Here we can configure the beacon with UUID major and minor numbers.
Gimbal Mode: This is not based on iBeaecon specification, so any available sniffer wouldn’t be able to determine the configuration.

The mode is specified by selecting Beacon type in Gimbal manager tool for the selected Beacon.

If you look at from standard’s perspective, the gimbal mode seems absurd as this assumes that we have all infrastructure investment in Gimbal products. What if we already have a few non-Gimbal beacons, or we might already have invested in some tools based on iBeacon advertisements. But actually, it looks quite popular based on its perceived security benefits.

It is rumored that iBeacon type is not available for all Gimbal beacons especially for series 10, but there are other opinions too. Let me set my hands on series 10 beacon and then we can talk. It is also limited based on Firmware versions.

Configuration Mode for Gimbal’s Beacons
As you insert the battery, gimbal’s beacon startsup and enters in configuration mode. It can then be applied with a configuration created using Gimbal Manager. You will need to download Gimbal’s app for updating this configuration.

Gimbal’s Beacon’s Public & Private Visibility
Gimbal’s beacons can also be set in either of public or private modes. In private mode, the beacon is available only for you account.

Visibility for Apps using Gimbal SDK
The application based on Gimbal’s SDK can also have different visibility modes. They are as follows:

Private
Authorized
Public

For authorization, the app uses OAuth to authrorize with Gimbal Manager.

Beacon Hardware Decisions

Beacon Hardware Decision

Beacon provides us the ability to create applications which are location aware, context aware and personalized. This is the data which cannot be faked.

Beacons come in different shapes and sizes. Since they all need to be based on iBeacon specification, it seems that we can be provider agnostic here. Had this been the case, then you could just get the cheapest of all as every beacon would function exactly the same. This is the reason, beacon providers keep trying differentiating themselves providing certain value additions. As a solution developer based on beacon technology, this would make our job a little difficult. I thought this would be a good idea to discuss various beacon hardware from different manufacturers, and see what I personally like / dislike in them.

Just remember this acronym: S³C²RIP. This is Security, Specs, SDK, Configuration, Charging, Range, Indoor / Outdoor and Price. Obviously, they are not in the order of their importance but all of them are required for selecting a particular beacon provider.

Who would be making this decision?

I think the decision would be done by the solution providers. If I am developing a solution for a retail, this decision would be made by me. This can be an add-on for the app I am providing to the retail store. I can charge my clients the add-on cost for proximity based services.

The shops on the main street (Mom & Pop shops), can also take advantage of beacons by advertising coupons for people who are already in their close proximity. In this case, the decision would still be made by solution provider. Now he can charge the shop for the coupons advertised.

Beacon Providers

There are a number of beacon providers. Estimote is hands down the most famous iBeacon tag provider. Please remember that this is not all-inclusive list, plus the order of provider in the list, is arbitrary. I just wanted to list the most common ones as the list is ever growing.

How to decide

Specification

You can find three different specifications on the web. They are iBeacon (Apple), AltBeacon (Radius Network) and Placedge Beacon (Samsung). AltBeacon promotes itself as open specification but I was not able to find any beacon hardware other than ones from Radius Network itself. I guess we need an open specification but AltBeacon doesn’t seem to have flown in public eyes.

Security

Security is always a relative concept. I think there are two important issues for iBeacon security. The are securing beacon configuration and broadcast advertisements.

The manufacturer generally provides an app to configure an iBeacon. The configuration involves setting the values for Proximity UUID, major and minor numbers. The question would be, can I get any app from market, which would allow me to configure your iBeacons. If yes, we have an issue. For a store example, this would mean that no customer would be getting the notifications about the wonderful things they can enjoy in their current visit.

Configuration App for your device

iBeacons require configuration for setting the values for Proximity UUID, major and minor numbers. The manufactures generally provide an app for this configuration. I have generally seen organization adopting an eco-system, like apple or android. If all of my employees are using android, and there is no android configuration app, there is a problem.

SDK for App Development

As an app developer, we don’t want to target specially iOS or Android operating system individually. As a matter of fact, we don’t generally have a choice for commercial apps if we are serious about selling our app in the market. We have to develop targeting multiple operating systems. Beacon providers include SDK package for making it easier for app development. We need to check whether there is a SDK for our targeting platform.

Charging

Beacons are based on Bluetooth Low Energy. So they don’t drain a lot of battery. The advertised battery life of Estimote IBeacon is 3 years. Motorola’s MPact beacon has its battery life based on the mode. The minimum battery life is in MPact mode (1 year).

But they still need power for their operations. Most of the beacons in the market use Coin batteries. One Radius network’s beacon hooks up to USB charger. This should be great for your testing and development. It is difficult to find USB hooks in the actual environments.

It would be very seldom that you would need to change the batteries of a beacon. But how to check if the beacon needs a replacement battery now? Definitely, we shouldn’t have to wait until we stop getting the notification data for a few days, and then just change the battery. Beacon providers have tried to make our life easier a little here e.g. MPact mode for Motorola’s MPact beacon provides visibility into battery status remotely.

Range

Beacon signals are Bluetooth Low Energy signals. It has a range of upto 70 meters (230 ft.). I have noticed that some beacon providers just want to play it safe and advertise a range lesser than 70 meters. But usually, you would see something like this:

Indoor / Outdoor Operations

Buying decision should also depend on the conditions we are planning to keep the beacons in. We might need to place beacons outside. In this case, they need to withstand outside conditions including sunlight, water and high / low temperature.

But do remember that Bluetooth has known limitation with water as it absorbs radio signals. So you might experience dropping the range of operations during the rain specially because of reduced signal strength.

Estimote advertises its Beacons to be water resistant. They are in silicone casing. The ideal operating temperature lies within -10°C (14°F) and 60°C (140°F).

Price & Packaging

One thing I specially dislike by many beacon providers is how they package it. The package includes different counts of iBeacons. You can buy them in the package of three, five, ten or more. I am a developer and just need one to develop an app. Why should I buy a pack for my testing. Estimote pack gets to approximately $100.

Beacons are simple devices based on Bluetooth Smart technology. As a beacon, the minimum requirement is just to support advertisements. This should make hardware really cheap. As discussed above, generally, they come in packages containing more than one beacons.

But proximity solutions wouldn’t be as cheap.

Note

Estimote Beacons are also equipped with accelerometer and temperature sensors. The accelerometer is used to detect motion. I have seen other providers also including additional sensors e.g. iBKS105 has a hall sensor, which makes the beacon enter in configuration mode. Adding new hardware increases cost of manufacturing. If my beacon has just enough hardware as I need to develop my solution, we should less care for the additional sensors it has.

Bluetooth Smart Beacon Specifications

Why Different Beacon Specifications?

Apple released iBeacon with iOS 7. iOS is a very big segment of smart phone users but there is a big market for android and other OSes as well. As apple made it difficult for the iBeacon manufacturers for providing SDKs for Android, they came up with alternate specifications as a workaround. One such example is AltBeacon from Radius Network. You might hear a general term for beacons now, they are called BLE Beacons or Proximity Beacons.It must be remembered that all iBeacons are also BLE (Bluetooth Low Energy) beacons as they are based on the same technology.

There are a number of beacon specifications. The most popular specifications seems to be the following:

iBeacon (from Apple)
AltBeacon (from Radius Network)
Placedge (from Samsung)

iBeacon

iBeacon is the first beacon specification. The specification is provided by Apple. When you are buying your beacon hardware, the following shows that the hardware is based on iBeacon specification.

The specification requires the beacons to transmit beacon advertisement every 100 ms. It’s for manufacturers, so as developers, we don’t have to worry about it as much.

AltBeacon:

This specification was introduced by Radius Networks, a provider of various beacon hardware. The specification was released in July, 2014. It has been released as an open specification.

AltBeacon allows payload space to be used for additional useful information. iBeacon payload can contain no data other than 3 identifiers. I could find no other beacons from outside manufacturer supporting this specification. Radius Network’s beacons obviously support this.

Placedge Beacon:

The specification was introduced by Samsung earlier this month (November, 2014).

Radius Network has a beacon supporting the specification.

There are also restrictions on the software side. Placedge SDK will only work with those beacons which are registered with Samsung with Placedge Console or Server API. They have a partner program for this purpose. Additionally the SDK is only for Android OS running 4.3 (Jelly Beans) or above. It is not very clear whether hardware manufacturers would take care of this registration or Solution providers would bear the pain of this registration. Since they need the partnership to use SDK anyway, it is not as bad than it first sounds. But this is certainly more restricted than Apple.

Multi Beacons

MultiBeacon is an idea to provide advertisement, based on multiple specifications simultaneously, from the same device. It is like those dual mode devices, which support multiple wireless protocols. This is to reduce market fragmentation. As a solution provider, it would be a nightmare to provide different systems for Apple and non-apple customers, hence multi-beacons.

Motorola’s MPact platform uses a combination of Wi-Fi and beacons for indoor location. In addition to iBeacon mode, it also supports Batter Save and MPact modes. The details can be found here. Additionally, it provides tools for providing this data for your analysis. It also supports multiple beacon modes including iBeacon.

Even when you see a sign like this in future, it is better to check what are the actual specifications supported as there are more than two specifications now, and the very specification might be missing, which your users might be needing.