How to Contribute to HBase and Hadoop 2.0

In case you haven’t heard, Hadoop 2.0 is on the way! There are loads more new features than I can begin to enumerate, including lots of interesting enhancements to HDFS for online applications like HBase. One of the most anticipated new features is YARN, an entirely new way to think about deploying applications across your Hadoop cluster. It’s easy to think of YARN as the infrastructure necessary to turn Hadoop into a cloud-like runtime for deploying and scaling data-centric applications. Early examples of such applications are rare, but two noteworthy examples are Knitting Boar and Storm on YARN. Hadoop 2.0 will also ship a MapReduce implementation built on top of YARN that is binary compatible with applications written for MapReduce on Hadoop-1.x.

The HBase project is rearing to get onto this new platform as well. Hadoop2 will be a fully supported deployment environment for HBase 0.96 release. There are still lots of bugs to squish and the build lights aren’t green yet. That’s where you come in!

Squash your first HBase + Hadoop2 bug

Start by grabbing the HBase source if you don’t have it already.

$ git clone git://git.apache.org/hbase.git
$ cd hbase

To build it, you’ll also need a modern version of maven installed. Installation is platform dependent.

To build HBase against Hadoop2, you’ll need to enable the appropriate profile. Use the -Dhadoop.profile=2.0argument to enable it. The first time you build HBase will take a while as you’ll need to download a bunch of dependencies. Maven takes care of this for you.

$ mvn -Dhadoop.profile=2.0 clean package -DskipTests
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

If everything went well, you now have a working build of HBase vs Hadoop2. Now go back to Jenkins and pick your favorite regression. TestImportTsv looks interesting. HBase uses the Surefire plugin to handle tests, which makes it easy to run just a single test.

$ mvn -Dhadoop.profile=2.0 test 
-Dtest=org.apache.hadoop.hbase.mapreduce.TestImportTsv#testBulkOutputWithAnExistingTable
...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
...
*boom*

Now that you’ve reproduced the bug, dive into the source, hack up a solution, and rebuild and rerun the test. All the usual maven build phases apply, just be sure you always include the -Dhadoop.profile=2.0 in your build commands. Once you have a working solution, find the relevant JIRA (ie, HBASE-8454), generate a patch, and post your fix.

Congratulations! You’re now an HBase contributor :D

What’s that? You’re only just now getting to read this post and TestImportTsv has already been fixed? Don’t worry, there’s plenty more issues to tackle.

Categorized by :
HBase YARN

Leave a Reply

Your email address will not be published. Required fields are marked *

If you have specific technical questions, please post them in the Forums

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Join the Webinar!

YARN Ready – Integrating to YARN using Slider (part 2 of 3)
Thursday, August 7, 2014
12:00 PM Eastern / 9:00 AM Pacific

More Webinars »

Integrate with existing systems
Hortonworks maintains and works with an extensive partner ecosystem from broad enterprise platform vendors to specialized solutions and systems integrators.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.