In this tutorial, we are going to configure IPython notebook with Apache Spark on YARN in a few steps.
IPython notebook is an interactive Python shell which lets you interact with your data one step at a time and also perform simple visualizations.
IPython notebook supports tab autocompletion on class names, functions, methods, variables. It also offers more explicit and colour-highlighted error messages than the command line python shell. It provides integration with basic UNIX shell allowing you can run simple shell commands such as cp, ls, rm, cp, etc. directly from the IPython. IPython integrates with many common GUI modules like PyQt, PyGTK, tkinter as well wide variety of data science Python packages.
This tutorial is a part of series of hands-on tutorials to get you started with HDP using Hortonworks sandbox. Please ensure you complete the prerequisites before proceeding with this tutorial.
Note: This tutorial only works with HDP 2.4 Sandbox or earlier.
To begin, login in to Hortonworks Sandbox through SSH:The default password is
Now let’s configure the dependencies by typing in the following command:
yum install nano centos-release-SCL zlib-devel \ bzip2-devel openssl-devel ncurses-devel \ sqlite-devel readline-devel tk-devel \ gdbm-devel db4-devel libpcap-devel xz-devel \ libpng-devel libjpg-devel atlas-devel
IPython has a requirement for Python 2.7 or higher. So, let’s install the “Development tools” dependency for Python 2.7
yum groupinstall "Development tools"
Now we are ready to install Python 2.7.
yum install python27
Now the Sandbox has multiple versions of Python, so we have to select which version of Python we want to use in this session. We will choose to use Python 2.7 in this session.
Next we will install pip using
pip makes it really easy to install the Python packages. We will use
pip to install the data science packages we might need using the following commands:
pip install numpy scipy pandas \ scikit-learn tornado pyzmq \ pygments matplotlib jsonschema
pip install jinja2 --upgrade
Finally, we are ready to install IPython notebook using
pip using the following command:
pip install jupyter
Since we want to use IPython with Apache Spark we have to use the Python interpreter which is built with Apache Spark,
pyspark, instead of the default Python interpreter.
As a first step of configuring that, let’s create a IPython profile for
ipython profile create pyspark
Next we are going to create a shell script to set the appropriate values every time we want to start IPython.
Create a shell script with the following command:
Then copy the following lines into the file:
#!/bin/bash source /opt/rh/python27/enable IPYTHON_OPTS="notebook --port 8889 \ --notebook-dir='/usr/hdp/current/spark-client/' \ --ip='*' --no-browser" pyspark
Save and exit your editor.
Finally we need to make the shell script we just created executable:
chmod +x start_ipython_notebook.sh
We need to forward the port
8889 from the guest VM, Sandbox to the host machine, your desktop for IPython notebook to be accessible from a browser on your host machine.
Open the VirtualBox App and open the settings page of the Sandbox VM by right clicking on the Sandbox VM and selecting settings.
Then select the networking tab from the top:
Then click on the port forwarding button to configure the port. Add a new port configuration by clicking the
+ icon on the top right of the page.
Input a name for application, IP and the guest and host ports as per the screenshot below:
OK to confirm the change in configuration.
Now we are ready to test IPython notebook.
Execute the shell script we created before from the sandbox command prompt using the command below:
Now, open a browser on your host machine and navigate to the URl http://127.0.0.1:8889 and you should see the screen below:
Voila! You have just configured IPython notebook with Apache Spark on you Sandbox.
In the next few tutorials we are going to explore how we can use IPython notebook to analyze and visualize data.