You’ll make an IoT Weather Station with a Raspberry Pi and Sense HAT. Additionally, you’ll add on data analytics to this IoT Weather Station Platform with Connected Data Architecture communication between the MiNiFi, HDF Sandbox and HDP Sandbox.
- Downloaded Latest HDF Sandbox for VirtualBox, etc
- Set RAM to 12GB for Guest VM or Native Docker
- Downloaded and Installed Latest Raspbian OS onto Raspberry Pi
- If you need help installing Raspbian OS onto the Raspberry Pi, refer to Appendix A.
- Enabled Connected Data Architecture:
- Step 1: Setup Raspberry Pi Weather Station Node
- Step 2: Configure Bridged Adapter Network for VirtualBox
- Step 3: Start HDF Sandbox and Setup NiFi Site-To-Site
- Further Reading
- Appendix A: Install Raspbian OS onto Raspberry Pi
- Appendix B: Reset Raspberry Pi Time and Date for your Region
Step 1: Setup Raspberry Pi Weather Station Node
1.1: Connect Sense HAT Hardware to Raspberry Pi
1. Connect the Sense HAT’s 40 female pins to the Raspberry Pi’s 40 male pins.
Figure 1: Sense HAT 40 female pin headers
Figure 2: Raspberry Pi 40 male pin headers
Figure 3: Sense HAT female pins connected to Raspberry Pi male pins
Note: If you haven’t installed Raspbian OS on your Raspberry Pi, refer to Appendix A.
1.2: Power the Raspberry Pi and Establish Internet Connection
Plug into the Raspberry Pi a micro USB (could be phone charger) connector to provide power and ethernet to provide internet access.
Figure 4: Raspberry Pi Ethernet Cable Connected for Internet Access
1.3: SSH into the Raspberry Pi
You’ll use Adafruit’s Pi Finder to discover your Raspberry Pi’s IP address, so you can access it remotely, this approach works best in a home network.
1. Download Adafruit’s Pi Finder for your appropriate OS at Adafruit-Pi-Finder Latest Releases
2. Open Raspberry Pi Finder and Click Find My Pi!:
Figure 5: Pi Finder GUI
Pi Finder – used to detect Raspberry Pi IP address in a home setting
3. Results include the IP address, SSH User and SSH Password of your Raspberry Pi:
Figure 6: Pi Finder Found My Pi
4. SSH into the Pi from your laptop by pressing the Terminal button.
Figure 7: Pi Raspbian OS CLI
1.4: Install the Sense HAT Software onto Raspberry Pi
Download and install the Sense HAT Software using the following command:
sudo apt-get update sudo apt-get install sense-hat sudo pip3 install pillow
Now you have the Sense HAT software library, so you can create a program to retrieve sensor readings, you will utilize it in the next tutorial.
1.5: Install MiNiFi Java Agent onto Raspberry Pi
In this section, you install Java 8 and JDK 1.8 onto the Raspberry Pi because it is required to run MiNiFi.
Download and Install Java 8 and JDK1.8:
sudo apt-get update && sudo apt-get install oracle-java8-jdk
Note: the install will take approximately 10 minutes depending on Raspbian OS resources being used.
1. Download MiNiFi Java Agent from Apache nifi minifi Downloads under section Releases -> MiNiFi (Java) -> Binaries.
2. Click on
minifi-[latest-version]-bin.zip, then download MiNiFi from any of the links provided onto your computer.
Figure 8: Downloading MiNiFi
3. Use Pi Finder’s Upload button to transport MiNiFi application onto your
Raspberry Pi. Select
minifi-[latest-version]-bin.zip and click Open.
Figure 9: Uploading MiNiFi to Raspberry Pi
4. Use the Pi Finder’s Terminal button to enter the Raspberry Pi and then Unzip the MiNiFi project.
A MiNiFi Agent is installed onto the Raspberry Pi. We’ll explain more about the MiNiFi Agent in the next tutorial.
1.6: Download MiNiFi Toolkit onto your Computer
In this section, you download MiNiFi toolkit onto your computer because it is needed to convert NiFi flow to MiNiFi flow format. In the next tutorial, you will build the MiNiFi flow in NiFi.
1. Download MiNiFi Toolkit from Apache nifi minifi Downloads under section Releases -> MiNiFi Toolkit Binaries -> [latest-version] – Compatible with MiNiFi Java [latest-version].
2. Click on
minifi-toolkit-[latest-version]-bin.zip then download MiNiFi Toolkit from any of the links provided onto your computer.
3. Go to the location where MiNiFi was downloaded and Unzip MiNiFi Toolkit
using your favorite decompression software:
Figure 10: Extracting MiNiFi Toolkit
Now MiNiFi Toolkit is available for use in the next tutorial.
1.7: Calibrate Raspberry Pi Timezone
Why is it important to calibrate the timezone on the Raspberry Pi?
In order for your system to have the correct time and date, the timezone needs to be calibrated. An area where this configuration is important is in the next tutorial when you create the Python script that draws timestamp for each sensor readings.
SSH into the Raspberry Pi using Adafruit’s Pi Finder Terminal button.
- raspi-config is used for changing OS configurations and will be used to recalibrate the current date/time for your timezone
Figure 11: raspi-config main menu
4. Internationalisation Options. Press “Enter” on keyboard.
I2 Change Timezone.
Figure 12: Internationalisation Options Menu
4. Select your appropriate
- Ex: US
Figure 13: Geographic area Selection Items
5. Select your appropriate
- Ex: Pacific Ocean
Figure 14: Time Zone Selection Items
6. You are brought back to the menu. Select
<Finish>. Your new calibrated time should display:
Figure 15: Time Zone Calibrated
Step 2: Configure Bridged Adapter Network for VirtualBox
VirtualBox will be shown as an example in the tutorial flow, but if you are
a VMware user, the settings will be similar. Open VirtualBox Manager.
If your Guest VM is running, it will need to be stopped.
1. Click the Settings gear.
2. In the window that opens, navigate to the Network tab.
3. In the Attached to field, select from the dropdown menu, Bridged Adapter.
You can keep the default setting for the Name of the Bridged Network.
Figure 16: Setting VirtualBox Guest VM to Bridged Adapter
Note: Verify your System RAM (Base Memory) is at least 12288 MB (12GB)
Figure 17: Verifying RAM allocation is 12GB
Step 3: Configure NiFi via HDF’s Ambari
1. Login to Ambari at
Note: The user/password is
You will configure NiFi Site-To-Site protocol by exposing an IP address and a socket port, so external NiFi nodes or MiNiFi agents can connect to NiFi master node.
Advanced NiFi-Properties in Ambari Config Settings for NiFi. Update the following configurations similar to the image below:
1. Enter NiFi Service in Ambari Stack
2. Enter NiFi Configs
3. Filter search for nifi.remote
Figure 18: Update NiFi Config for Site-to-Site
4. Insert your
<guest vm ip address> in nifi.remote.input.host. The image above shows “10.11.7.17”, which was the IP address assigned by the router in an office network, for a home network, we can see in the image below that the IP address was changed to “192.168.2.10”.
<guest vm ip address>for linux can be found with the terminal command:
hostname -I. Yet, when you open the sandbox, we present a welcome screen to you that displays the IP address of your guest vm from bridged adapter network. Since we are using VirtualBox, there is a typo “For VMware:”, which actually shows the guest vm ip address we want from our VirtualBox VM. VMware Guest VM welcome screen will have the IP address for the VMware Guest VM.
Figure 19: Get Guest VM IP Address from Sandbox Welcome Screen
5. Verify nifi.remote.input.http.enabled checked
15500 in nifi.remote.input.socket.port.
7. Save the configuration. Write in Notes
Configured NiFi for Socket Site-To-Site
Now NiFi is configured for Socket Site-To-Site protocol. If you encounter issues deploying MiNiFi to NiFi flow, it could be because the value nifi.remote.iput.host changed or nifi.remote.input.socket.port you chose is already being used. Of course there are other reasons for issues, but these two are ones to be mindful of.
Note: Now that you know where to find the Guest VM Sandbox IP address, set the mapping for it to the hostnames: “[ip-adr] sandbox-hdf.hortonworks.com sandbox-hdp.hortonworks.com”. Refer to “Learning the Ropes of HDP Sandbox“.
3.3: Restart NiFi
Restart NiFi from Ambari with the orange restart button for the changes to take effect.
3.4: Add GeoLite2 database to HDF Sandbox CentOS
Add the GeoLite2 to HDF Sandbox CentOS, which is a database filled with Public IP Addresses mapped to geographic insights.
1. Access HDF Web Shell Client at
sandbox-hdf.hortonworks.com:4200. User/Password is
Figure 20: HDF Web Shell
Note: You will be prompted to change the password if this is your first time logging into the Sandbox.
2. Create directory for GeoFile. Change permissions of the directory to ensure NiFi has access.
mkdir -p /sandbox/tutorial-files/820/nifi/input/GeoFile chmod 777 -R /sandbox/tutorial-files/820/nifi/
3. Download GeoLite2-City.mmdb to specified location GeoEnrichIP looks:
cd /sandbox/tutorial-files/820/nifi/input/GeoFile wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz tar -zxvf GeoLite2-City.tar.gz
4. Get the full pathname to GeoLite2-City.mmdb:
cd GeoLite2-City_[date-updated] ls pwd
Figure 21: Path to Geolite DB Lookup Table “GeoLite2-City.mmdb”
Note down the folder name that GeoLite2-City.mmdb is located in on your system. According to the image above, the full pathname is:
Note: GeoLite2-City_[date-updated] according to the image above is GeoLite2-City_20170704. However, at the time you download the GeoLite DB file, your [date-updated] will most likely be different than 20170704, so if needed, update the full pathname.
This full pathname will be used in a later tutorial when you extend the NiFi Flow to pull in geographic insights.
Congratulations! You know how to setup your own IoT Weather Station using the Raspberry Pi, Sense HAT, MiNiFi, HDF Sandbox and HDP Sandbox. You are also familiar with how to install MiNiFi onto the Raspberry Pi. You also have begun setting up Connected Data Architecture between HDF and HDP SB nodes in a Docker Network. In the next tutorials, you’ll focus on data preprocessing, data storing into a noSQL datastore and analyzing the data in real-time as it saves to the datastore.
- Learn to Install Raspbian OS through the official Raspberry Pi Software Guide
- Learn to explore the Pi Finder through The Adafruit Raspberry Pi Finder Guide
- Explore the functionality of each pin through Raspberry Pi Pinout Diagram
Appendix A: Install Raspbian OS onto Raspberry Pi
For users who need help installing Raspbian OS onto their Raspberry Pi, we have provided a step-by-step procedure. First you insert the microSD card into your computer, then download the Raspbian Image OS to your computer. We will walk you through two approaches for creating the bootable Raspbian OS. Approach 1 uses Etcher SD Card Image Utility GUI program whereas Approach 2 uses CLI.
Insert microSD Card into Computer
1. Connect MicroSD to microSD Adapter.
Figure 22: MicroSD on left and microSD Card Adapter on right
2. Insert the microSD Adapter into the computer.
Figure 23: microSD Adapter Inserted into Computer
Download Raspbian OS Image
3. Download Rasbpian Jessie Lite OS Image onto your host machine.
4. Unzip the OS Image.
Approach 1: Create a Bootable Raspbian OS on microSD Card with Etcher.io
You will create a Raspbian bootable OS on microSD card using etcher.io graphic install wizard instead of the command line.
1. Download and install Etcher SD card image utility from etcher.io
2. Use Etcher to transfer the Raspbian image to the location of your SD card. Select Image of Raspbian OS from the download location, then Select Drive (your microSD Card may already be selected by default) and choose Flash! to create a bootable Flash Image on the microSD.
Figure 24: Etcher Dashboard to Create a Bootable OS on microSD
Figure 25: Flash Complete, Bootable OS Now Created
Once the operation completes, Etcher automatically unmounts the SD card and is safe to eject.
Approach 2: Create a Bootable Raspbian OS on microSD with CLI (MAC/Linux)
1. See a list of all devices that are mounted on laptop using command:
2. Note down the device path listed next to the volume, look for the most recent volume added, it’ll probably have the path /Volumes/BOOT under Mounted On column.
3. Open Disk Utility, select SD card, then press Unmount, so we can write to the entire card.
Figure 26: MAC Disk Utility to Unmount Device for Writing to it
4. Head to terminal, in the Downloads folder where the Raspbian OS is located, run the DD command to write a bootable Raspbian OS onto micro SD card:
sudo dd bs=1m if=2017-02-16-raspbian-jessie-lite.img of=/dev/rdisk2
Note: Explanation of three arguments used in dd: bs = block size, if = location of raspbian input file, of = location of peripheral device output file. Notice how in the of= argument, SD card volume changes from disk2s1 to rdisk2 to ensure the entire SD card is overloaded, not just the partition.
The DD operation will take 1 to 5 minutes until completion.
Figure 27: Progress of Creating Bootable OS on microSD
After the dd operation completes, you should see the Raspbian bootable OS successfully transferred over to the SD card.
5. To setup a headless raspberry pi, ssh can be enabled by placing a file named ssh onto the boot partition’s base directory, the following is an example of how that would look:
boot jmedel$ pwd boot jmedel$ touch ssh boot jmedel$ ls COPYING.linux bcm2708-rpi-cm.dtb cmdline.txt fixup_x.dat ssh LICENCE.broadcom bcm2709-rpi-2-b.dtb config.txt issue.txt start.elf LICENSE.oracle bcm2710-rpi-3-b.dtb fixup.dat kernel.img start_cd.elf bcm2708-rpi-b-plus.dtb bcm2710-rpi-cm3.dtb fixup_cd.dat kernel7.img start_db.elf bcm2708-rpi-b.dtb bootcode.bin fixup_db.dat overlays start_x.elf
Figure 28: Create SSH file to Enable SSH Access to Raspberry Pi
Note: the path to the SD card is
touch sshcreates a new file.
ls -ltrverifies new file was created.
6. Eject the microSD card Adapter and remove it from your laptop. Insert the microSD card into the micro SD card slot of the Raspberry Pi.
Figure 29: MicroSD Inserted into Raspberry Pi
7. Connect ethernet cable to the Raspberry Pi to give it internet access, connect the 5V for power and the Pi should start up.
Figure 30: Raspberry Pi Ethernet Cable Connected for Internet Access
The Pi’s default login credentials:
username/password = pi/raspberry
Note: you will need the password for ssh access to the Raspberry Pi.