Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
HDF > Develop Data Flow & Streaming Applications > Hello World

NiFi in Trucking IoT on HDF

Run NiFi in the Demo

cloud Ready to Get Started?

DOWNLOAD SANDBOX

Introduction

Let’s walk through NiFi’s place in the demo.

Outline

Environment Setup

We will be working in the trucking-IoT project. If you have the latest Hortonworks DataFlow (HDF) Sandbox installed, then the demo comes preinstalled. If not or you do not already have it setup, then refer to Setup Demo on existing HDF/HDP.

Your HDF Sandbox should be powered on using your favorite platform.

Deploy the NiFi DataFlow

Let’s activate the NiFi dataflow, so it will process the simulated data and push the data into Kafka Topics. Open NiFi at sandbox-hdf.hortonworks.com:9090/nifi.

Drag and drop the components template icon nifi_template onto the NiFi canvas. Select Trucking IoT Demo, then press ADD. Deselect the dataflow, just click anywhere on the canvas.

In the Operate Pallete with the hand point upward, expand it if it is closed, click on the gear icon then click on Controller Services gear icon. In Controller Services, click on the Lighting Bolt to the right of HortonworksSchemaRegistry, for Scope, select Service and referencing componen…, then press ENABLE. Then CLOSE. All the Controller Services should be enabled like below:

controller-services-en

Note: If any of your services are disabled, you can enable them by clicking on the lightning bolt symbol on the far right of the table. Controller Services are required to be enabled to successfully run the dataflow.

Let’s select the entire dataflow. Hold command or ctrl and A and the whole dataflow will be selected. In the Operate Pallete, click on the start button start-button and let it run for 1 minute. The red stop symbols red-symbol at the corner of each component in the dataflow will turn to a green play symbol green-symbol. You should see the numbers in the connection queues change from 0 to a higher number indicating that the data is being processed.

You should see a picture similar to below:

dataflow

Let’s analyze what actions the processors taking on the data via NiFi’s Data Provenance:

Unselect the entire dataflow then right click on GetTruckingData: Generates data of two types: TruckData and TrafficData. Click Data Provenance.

GetTruckingData

A table with provenance events will appear. An event illustrates what type of action the processor took against the data. For GetTruckingData, it is creating sensor data in two categories as one stream. Choose an event with 20 bytes to see TrafficData or greater than or equal to 98 bytes to see TruckData.

data-provenance

  • TruckData: Data simulated by sensors onboard each truck.

TruckData

  • TrafficData: Data simulated from traffic congestion on a particular trucking route.

TrafficData

You can check the data provenance at each processor to get a more in-depth look at the steps NiFi is performing to process and transform the two types of simulated data. Here is a flow chart to show the steps:

nifi-flow-chart

Next: Building a NiFi DataFlow

Now that we know how NiFi fits into the data pipeline of the demo and what kind of transformations on the data is performing, let’s dive into configuring processors to see how the dataflow is constructed.

User Reviews

User Rating
0 No Reviews
5 Star 0%
4 Star 0%
3 Star 0%
2 Star 0%
1 Star 0%
Tutorial Name
NiFi in Trucking IoT on HDF

To ask a question, or find an answer, please visit the Hortonworks Community Connection.

No Reviews
Write Review

Register

Please register to write a review

Share Your Experience

Example: Best Tutorial Ever

You must write at least 50 characters for this field.

Success

Thank you for sharing your review!