Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics, offering information and knowledge of the Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
HDP > Develop with Hadoop > Apache Spark

Intro to Machine Learning with Apache Spark and Apache Zeppelin

cloud Ready to Get Started?



Spark MLlib Logo

In this tutorial, we will introduce you to Machine Learning with Apache Spark. The hands-on lab for this tutorial is an Apache Zeppelin notebook that has all the steps necessary to ingest and explore data, train, test, visualize, and save a model. We will cover a basic Linear Regression model that will allow us perform simple predictions on a sample data. This model can be further expanded and modified to fit your needs. Most importantly, by the end of this tutorial, you will understand how to create an end-to-end pipeline for setting up and training simple models in Spark.



Tutorial Details

Machine Learning models can be applied to accomplish a variety of tasks from Classification, to Collaborative Filtering, Clustering, and Regression.

We start with Linear Regression as this is one one of the most common approaches for estimating unknown parameters after training on a known dataset. In this tutorial, we are training on a 2D dataset, so our Linear Regression model can be intuitively thought as curve fitting. With more parameters, or features, we can make interesting predictions, for example, what should be a price listing range for a house with three bedrooms, two baths, 20 years old, and in a specific zip code area. Using Linear Regression for pricing houses given a set of input parameters works surprisingly well provided a large enough sample dataset. To cover edge cases, however, other Machine Learning methods might have to be used such as Random Forests or Gradient Boosted Trees, which we will cover in detail in future tutorials.

There are multiple steps in this lab we will cover.

First, the dataset will consist of pairs of inputs and expected outputs. You may think of this as (x, y) pairs, with x being the input and y being the output. We will have approximately twenty points in our small training dataset.

Next, we will construct a Linear Regression pipeline where we specify the input and output columns, and the model we want to use (i.e. Linear Regression). Once the pipeline is created, we will use the fit function to fit the data to a model (i.e. train the model).

Third, we will summarize the model results focusing on a) Root Mean Square Error (RMSE) as another measure of differences between predicted by model and observed values (taking in account all the differences or residuals) with lower value indicating a better fit; b) R2 or R Squared, also called coefficient of determination or goodness of fit, with the higher values indicating better fit (on a 0 to 1 scale); and c) residuals that indicate difference between the observed value and the estimated value of the quantify of interest. With these snapshot measures, a Data Scientist can quickly understand model fitness and compare different trained models to choose the best one.

Fourth, with the trained model we will make predictions and see how individual predictions compare to original (expected) values.

Finally, we will graph the model as a straight line overlaid over the original training data points. This will give you a quick visual snapshot on the fitness of the model.

To summarize, we will be 1) setting up a two dimensional dataset, 2) creating a Linear Regression model pipeline, 3) summarizing the model training, 4) predicting output using the model, 5) and finally visualizing the results.

Environment Setup

Option 1: Setup Hortonworks Data Cloud (HDCloud) on AWS

This option is ideal if you want to experience a production-ready multi-node cluster in a cloud.

See the Getting Started with HDCloud tutorial for details.

Option 2: Download and Setup Hortonworks Data Platform (HDP) Sandbox

This option is optimal if you prefer to run everything in local environment (laptop/PC).

Keep in mind, that you will need 8GB of memory dedicated for the virtual machine, meaning that you should have at least 12GB of memory on your system.

2a. Download and Install HDP Sandbox 2.6

2b. Review Learning the Ropes of HDP Sandbox

Review Zeppelin Tutorial

If you are new to Zeppelin, review the following tutorial Getting Started with Apache Zeppelin

Notebook Preview

Before you start, here’s a preview of the notebook.

A dynamic preview (allowing code copy) can be found here.

Start the Tutorial

To begin the tutorial, import the Intro to Machine Learning notebook into your Zeppelin environment. (If at any point you have any issues, make sure to checkout the Getting Started with Zeppelin tutorial.)

On the Zeppelin home screen click Import note -> Add from URL and copy and paste the following URL:

Once your notebook is imported, you can open it from the Zeppelin home screen by clicking
Labs -> Spark 2.x -> Data Scientist -> Scala -> 101 - Intro to Machine Learning

Once the Intro to Machine Learning notebook is up, follow all the directions within the notebook to complete the tutorial.

Final words

We hope that you’ve been able to successfully run this first lab introducing basic, yet very common, Linear Regression model.

Next, if you would like to learn more on how to deploy machine learning models in production, checkout this tutorial on deploying machine learning models with structured streaming.

For a more advanced example, checkout our tutorial on Twitter sentiment analysis.

Finally, make sure to checkout our Data Science Starter Kit with pre-selected videos, tutorials, white papers and other content related to Data Science and Machine Learning.