newsletter

Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
HDF > Develop Data Flow & Streaming Applications > Real World Examples

Analyze IoT Weather Station Data via Connected Data Architecture

Deploy IoT Weather Station and Connected Data Architecture

cloud Ready to Get Started?

DOWNLOAD SANDBOX

Introduction

You’ll make an IoT Weather Station with a Raspberry Pi and Sense HAT. Additionally, you’ll add on data analytics to this IoT Weather Station Platform with Connected Data Architecture communication between the MiNiFi, HDF Sandbox and HDP Sandbox.

Prerequisites

Outline

Step 1: Setup Raspberry Pi Weather Station Node

1.1: Connect Sense HAT Hardware to Raspberry Pi

1. Connect the Sense HAT’s 40 female pins to the Raspberry Pi’s 40 male pins.

sense-hat-pins

Figure 1: Sense HAT 40 female pin headers

raspberry-pi-pins

Figure 2: Raspberry Pi 40 male pin headers

iot-device-pins-connected

Figure 3: Sense HAT female pins connected to Raspberry Pi male pins

Note: If you haven’t installed Raspbian OS on your Raspberry Pi, refer to Appendix A.

1.2: Power the Raspberry Pi and Establish Internet Connection

Plug into the Raspberry Pi a micro USB (could be phone charger) connector to provide power and ethernet to provide internet access.

power-ethernet_rpi

Figure 4: Raspberry Pi Ethernet Cable Connected for Internet Access

1.3: SSH into the Raspberry Pi

You’ll use Adafruit’s Pi Finder to discover your Raspberry Pi’s IP address, so you can access it remotely, this approach works best in a home network.

1. Download Adafruit’s Pi Finder for your appropriate OS at Adafruit-Pi-Finder Latest Releases

2. Open Raspberry Pi Finder and Click Find My Pi!:

find_my_pi

Figure 5: Pi Finder GUI

Pi Finder – used to detect Raspberry Pi IP address in a home setting

3. Results include the IP address, SSH User and SSH Password of your Raspberry Pi:

pi_finder_found_my_pi

Figure 6: Pi Finder Found My Pi

4. SSH into the Pi from your laptop by pressing the Terminal button.

successful_ssh_to_rpi

Figure 7: Pi Raspbian OS CLI

1.4: Install the Sense HAT Software onto Raspberry Pi

Download and install the Sense HAT Software using the following command:

sudo apt-get update
sudo apt-get install sense-hat
sudo pip3 install pillow

Now you have the Sense HAT software library, so you can create a program to retrieve sensor readings, you will utilize it in the next tutorial.

1.5: Install MiNiFi Java Agent onto Raspberry Pi

In this section, you install Java 8 and JDK 1.8 onto the Raspberry Pi because it is required to run MiNiFi.

Download and Install Java 8 and JDK1.8:

sudo apt-get update && sudo apt-get install oracle-java8-jdk

Note: the install will take approximately 10 minutes depending on Raspbian OS resources being used.

1. Download MiNiFi Java Agent from Apache nifi minifi Downloads under section Releases -> MiNiFi (Java) -> Binaries.

2. Click on minifi-[latest-version]-bin.zip, then download MiNiFi from any of the links provided onto your computer.

download_links_minifi

Figure 8: Downloading MiNiFi

3. Use Pi Finder’s Upload button to transport MiNiFi application onto your
Raspberry Pi. Select minifi-[latest-version]-bin.zip and click Open.

upload_minifi_to_rpi

Figure 9: Uploading MiNiFi to Raspberry Pi

4. Use the Pi Finder’s Terminal button to enter the Raspberry Pi and then Unzip the MiNiFi project.

unzip minifi-[latest-version]-bin.zip

A MiNiFi Agent is installed onto the Raspberry Pi. We’ll explain more about the MiNiFi Agent in the next tutorial.

1.6: Download MiNiFi Toolkit onto your Computer

In this section, you download MiNiFi toolkit onto your computer because it is needed to convert NiFi flow to MiNiFi flow format. In the next tutorial, you will build the MiNiFi flow in NiFi.

1. Download MiNiFi Toolkit from Apache nifi minifi Downloads under section Releases -> MiNiFi Toolkit Binaries -> [latest-version] – Compatible with MiNiFi Java [latest-version].

2. Click on minifi-toolkit-[latest-version]-bin.zip then download MiNiFi Toolkit from any of the links provided onto your computer.

3. Go to the location where MiNiFi was downloaded and Unzip MiNiFi Toolkit
using your favorite decompression software:

decompress_minifi_toolkit

Figure 10: Extracting MiNiFi Toolkit

Now MiNiFi Toolkit is available for use in the next tutorial.

1.7: Calibrate Raspberry Pi Timezone

Why is it important to calibrate the timezone on the Raspberry Pi?

In order for your system to have the correct time and date, the timezone needs to be calibrated. An area where this configuration is important is in the next tutorial when you create the Python script that draws timestamp for each sensor readings.

SSH into the Raspberry Pi using Adafruit’s Pi Finder Terminal button.

1. Type sudo raspi-config

  • raspi-config is used for changing OS configurations and will be used to recalibrate the current date/time for your timezone

raspi_config_menu

Figure 11: raspi-config main menu

2. Select 4. Internationalisation Options. Press “Enter” on keyboard.

3. Select I2 Change Timezone.

change_timezone

Figure 12: Internationalisation Options Menu

4. Select your appropriate Geographic area.

  • Ex: US

geographic_area

Figure 13: Geographic area Selection Items

5. Select your appropriate Time zone.

  • Ex: Pacific Ocean

time_zone

Figure 14: Time Zone Selection Items

6. You are brought back to the menu. Select <Finish>. Your new calibrated time should display:

output_calibrated_time

Figure 15: Time Zone Calibrated

Step 2: Configure Bridged Adapter Network for VirtualBox

VirtualBox will be shown as an example in the tutorial flow, but if you are
a VMware user, the settings will be similar. Open VirtualBox Manager.
If your Guest VM is running, it will need to be stopped.

1. Click the Settings gear.

2. In the window that opens, navigate to the Network tab.

3. In the Attached to field, select from the dropdown menu, Bridged Adapter.

You can keep the default setting for the Name of the Bridged Network.

set_vm_bridged_network

Figure 16: Setting VirtualBox Guest VM to Bridged Adapter

Note: Verify your System RAM (Base Memory) is at least 12288 MB (12GB)

verify_vm_ram_12gb

Figure 17: Verifying RAM allocation is 12GB

Step 3: Configure NiFi via HDF’s Ambari

1. Login to Ambari at sandbox-hdf.hortonworks.com:8080:

Note: The user/password is admin/admin

You will configure NiFi Site-To-Site protocol by exposing an IP address and a socket port, so external NiFi nodes or MiNiFi agents can connect to NiFi master node.

Head to Advanced NiFi-Properties in Ambari Config Settings for NiFi. Update the following configurations similar to the image below:

1. Enter NiFi Service in Ambari Stack

2. Enter NiFi Configs

3. Filter search for nifi.remote

advanced_nifi_properties

Figure 18: Update NiFi Config for Site-to-Site

4. Insert your <guest vm ip address> in nifi.remote.input.host. The image above shows “10.11.7.17”, which was the IP address assigned by the router in an office network, for a home network, we can see in the image below that the IP address was changed to “192.168.2.10”.

Note: <guest vm ip address> for linux can be found with the terminal command: hostname -I. Yet, when you open the sandbox, we present a welcome screen to you that displays the IP address of your guest vm from bridged adapter network. Since we are using VirtualBox, there is a typo “For VMware:”, which actually shows the guest vm ip address we want from our VirtualBox VM. VMware Guest VM welcome screen will have the IP address for the VMware Guest VM.

sandbox_vm_welcome

Figure 19: Get Guest VM IP Address from Sandbox Welcome Screen

5. Verify nifi.remote.input.http.enabled checked

6. Insert 15500 in nifi.remote.input.socket.port.

7. Save the configuration. Write in Notes Configured NiFi for Socket Site-To-Site

Now NiFi is configured for Socket Site-To-Site protocol. If you encounter issues deploying MiNiFi to NiFi flow, it could be because the value nifi.remote.iput.host changed or nifi.remote.input.socket.port you chose is already being used. Of course there are other reasons for issues, but these two are ones to be mindful of.

Note: Now that you know where to find the Guest VM Sandbox IP address, set the mapping for it to the hostnames: “[ip-adr] sandbox-hdf.hortonworks.com sandbox-hdp.hortonworks.com”. Refer to “Learning the Ropes of HDP Sandbox“.

3.3: Restart NiFi

Restart NiFi from Ambari with the orange restart button for the changes to take effect.

3.4: Add GeoLite2 database to HDF Sandbox CentOS

Add the GeoLite2 to HDF Sandbox CentOS, which is a database filled with Public IP Addresses mapped to geographic insights.

1. Access HDF Web Shell Client at sandbox-hdf.hortonworks.com:4200. User/Password is root/hadoop.

hdf_web_shell

Figure 20: HDF Web Shell

Note: You will be prompted to change the password if this is your first time logging into the Sandbox.

2. Create directory for GeoFile. Change permissions of the directory to ensure NiFi has access.

mkdir -p /sandbox/tutorial-files/820/nifi/input/GeoFile
chmod 777 -R /sandbox/tutorial-files/820/nifi/

3. Download GeoLite2-City.mmdb to specified location GeoEnrichIP looks:

cd /sandbox/tutorial-files/820/nifi/input/GeoFile
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
tar -zxvf GeoLite2-City.tar.gz

4. Get the full pathname to GeoLite2-City.mmdb:

cd GeoLite2-City_[date-updated]
ls
pwd

geolite_dbfile_path

Figure 21: Path to Geolite DB Lookup Table “GeoLite2-City.mmdb”

Note down the folder name that GeoLite2-City.mmdb is located in on your system. According to the image above, the full pathname is: /sandbox/tutorial-files/820/nifi/input/GeoFile/GeoLite2-City_20180605/GeoLite2-City.mmdb

Note: GeoLite2-City_[date-updated] according to the image above is GeoLite2-City_20170704. However, at the time you download the GeoLite DB file, your [date-updated] will most likely be different than 20170704, so if needed, update the full pathname.

This full pathname will be used in a later tutorial when you extend the NiFi Flow to pull in geographic insights.

Summary

Congratulations! You know how to setup your own IoT Weather Station using the Raspberry Pi, Sense HAT, MiNiFi, HDF Sandbox and HDP Sandbox. You are also familiar with how to install MiNiFi onto the Raspberry Pi. You also have begun setting up Connected Data Architecture between HDF and HDP SB nodes in a Docker Network. In the next tutorials, you’ll focus on data preprocessing, data storing into a noSQL datastore and analyzing the data in real-time as it saves to the datastore.

Further Reading

Appendix A: Install Raspbian OS onto Raspberry Pi

For users who need help installing Raspbian OS onto their Raspberry Pi, we have provided a step-by-step procedure. First you insert the microSD card into your computer, then download the Raspbian Image OS to your computer. We will walk you through two approaches for creating the bootable Raspbian OS. Approach 1 uses Etcher SD Card Image Utility GUI program whereas Approach 2 uses CLI.

Recommended Hardware:

Insert microSD Card into Computer

1. Connect MicroSD to microSD Adapter.

microsd_microsd_adapter

Figure 22: MicroSD on left and microSD Card Adapter on right

2. Insert the microSD Adapter into the computer.

insert_microsdAdater_laptop

Figure 23: microSD Adapter Inserted into Computer

Download Raspbian OS Image

3. Download Rasbpian Jessie Lite OS Image onto your host machine.

4. Unzip the OS Image.

Approach 1: Create a Bootable Raspbian OS on microSD Card with Etcher.io

You will create a Raspbian bootable OS on microSD card using etcher.io graphic install wizard instead of the command line.

1. Download and install Etcher SD card image utility from etcher.io

2. Use Etcher to transfer the Raspbian image to the location of your SD card. Select Image of Raspbian OS from the download location, then Select Drive (your microSD Card may already be selected by default) and choose Flash! to create a bootable Flash Image on the microSD.

etcher_dashboard

Figure 24: Etcher Dashboard to Create a Bootable OS on microSD

etcher_created_bootable_os

Figure 25: Flash Complete, Bootable OS Now Created

Once the operation completes, Etcher automatically unmounts the SD card and is safe to eject.

Approach 2: Create a Bootable Raspbian OS on microSD with CLI (MAC/Linux)

1. See a list of all devices that are mounted on laptop using command:

df

2. Note down the device path listed next to the volume, look for the most recent volume added, it’ll probably have the path /Volumes/BOOT under Mounted On column.

3. Open Disk Utility, select SD card, then press Unmount, so we can write to the entire card.

disk_utility_sd_unmount

Figure 26: MAC Disk Utility to Unmount Device for Writing to it

4. Head to terminal, in the Downloads folder where the Raspbian OS is located, run the DD command to write a bootable Raspbian OS onto micro SD card:

sudo dd bs=1m if=2017-02-16-raspbian-jessie-lite.img of=/dev/rdisk2

Note: Explanation of three arguments used in dd: bs = block size, if = location of raspbian input file, of = location of peripheral device output file. Notice how in the of= argument, SD card volume changes from disk2s1 to rdisk2 to ensure the entire SD card is overloaded, not just the partition.

The DD operation will take 1 to 5 minutes until completion.

dd_opearation_completion_result

Figure 27: Progress of Creating Bootable OS on microSD

After the dd operation completes, you should see the Raspbian bootable OS successfully transferred over to the SD card.

5. To setup a headless raspberry pi, ssh can be enabled by placing a file named ssh onto the boot partition’s base directory, the following is an example of how that would look:

boot jmedel$ pwd
boot jmedel$ touch ssh
boot jmedel$ ls
COPYING.linux           bcm2708-rpi-cm.dtb    cmdline.txt   fixup_x.dat   ssh
LICENCE.broadcom        bcm2709-rpi-2-b.dtb   config.txt    issue.txt     start.elf
LICENSE.oracle          bcm2710-rpi-3-b.dtb   fixup.dat     kernel.img    start_cd.elf
bcm2708-rpi-b-plus.dtb  bcm2710-rpi-cm3.dtb   fixup_cd.dat  kernel7.img   start_db.elf
bcm2708-rpi-b.dtb       bootcode.bin          fixup_db.dat  overlays      start_x.elf

Figure 28: Create SSH file to Enable SSH Access to Raspberry Pi

Note: the path to the SD card is /Volumes/boot. touch ssh creates a new file. ls -ltr verifies new file was created.

6. Eject the microSD card Adapter and remove it from your laptop. Insert the microSD card into the micro SD card slot of the Raspberry Pi.

microsd_inserted_to_rpi

Figure 29: MicroSD Inserted into Raspberry Pi

7. Connect ethernet cable to the Raspberry Pi to give it internet access, connect the 5V for power and the Pi should start up.

power-ethernet_rpi

Figure 30: Raspberry Pi Ethernet Cable Connected for Internet Access

The Pi’s default login credentials:

username/password = pi/raspberry

Note: you will need the password for ssh access to the Raspberry Pi.

User Reviews

User Rating
0 No Reviews
5 Star 0%
4 Star 0%
3 Star 0%
2 Star 0%
1 Star 0%
Tutorial Name
Analyze IoT Weather Station Data via Connected Data Architecture

To ask a question, or find an answer, please visit the Hortonworks Community Connection.

No Reviews
Write Review

Register

Please register to write a review

Share Your Experience

Example: Best Tutorial Ever

You must write at least 50 characters for this field.

Success

Thank you for sharing your review!