Apache Flink – Collections & Streams

While developing a streaming application, it is often necessary to use some inputs as collections. This can be specially useful for testing purposes. In this post we are trying to discuss how we can create a DataStream from a collection. After applying some operators on the stream, we are going to create an iterator from the elements of the stream.

StreamExecutionEnvironment supports creating streams from elements of a collection using fromElements method. Here we are using an integer collection. Here it returns DataStream[Int].

We are then using map operator to calculate double of the elements of each element. Finally DataStreamUtils is used to convert the stream to an iterator. We can now simply print out the elements of the iterator. Here is the output:

Output

It is important that DataStreamUtils is used after executing the stream operators. Otherwise, it returns in the following error on executing the stream using StreamExecutionEnvironment.

Download Code

https://github.com/msiddiqi/ApacheFlingStreamsAndIterators