Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
June 20, 2013
prev slideNext slide

How to Contribute to HBase and Hadoop 2.0

In case you haven’t heard, Hadoop 2.0 is on the way! There are loads more new features than I can begin to enumerate, including lots of interesting enhancements to HDFS for online applications like HBase. One of the most anticipated new features is YARN, an entirely new way to think about deploying applications across your Hadoop cluster. It’s easy to think of YARN as the infrastructure necessary to turn Hadoop into a cloud-like runtime for deploying and scaling data-centric applications. Early examples of such applications are rare, but two noteworthy examples are Knitting Boar and Storm on YARN. Hadoop 2.0 will also ship a MapReduce implementation built on top of YARN that is binary compatible with applications written for MapReduce on Hadoop-1.x.

The HBase project is rearing to get onto this new platform as well. Hadoop2 will be a fully supported deployment environment for HBase 0.96 release. There are still lots of bugs to squish and the build lights aren’t green yet. That’s where you come in!

Squash your first HBase + Hadoop2 bug

Start by grabbing the HBase source if you don’t have it already.

$ git clone git://
$ cd hbase

To build it, you’ll also need a modern version of maven installed. Installation is platform dependent.

To build HBase against Hadoop2, you’ll need to enable the appropriate profile. Use the -Dhadoop.profile=2.0argument to enable it. The first time you build HBase will take a while as you’ll need to download a bunch of dependencies. Maven takes care of this for you.

$ mvn -Dhadoop.profile=2.0 clean package -DskipTests
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------

If everything went well, you now have a working build of HBase vs Hadoop2. Now go back to Jenkins and pick your favorite regression. TestImportTsv looks interesting. HBase uses the Surefire plugin to handle tests, which makes it easy to run just a single test.

$ mvn -Dhadoop.profile=2.0 test 
 T E S T S
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv

Now that you’ve reproduced the bug, dive into the source, hack up a solution, and rebuild and rerun the test. All the usual maven build phases apply, just be sure you always include the -Dhadoop.profile=2.0 in your build commands. Once you have a working solution, find the relevant JIRA (ie, HBASE-8454), generate a patch, and post your fix.

Congratulations! You’re now an HBase contributor 😀

What’s that? You’re only just now getting to read this post and TestImportTsv has already been fixed? Don’t worry, there’s plenty more issues to tackle.



Vinisha says:

I tried command `mvn -Dhadoop.profile=2.0 clean package -DskipTests` but I am seeing following errors:

[ERROR] The build could not read 2 projects -> [Help 1]
[ERROR] The project org.apache.hbase:hbase-server:3.0.0-SNAPSHOT (/Users/vinishavyasa/work/hbase/hbase-server/pom.xml) has 1 error
[ERROR] ‘dependencies.dependency.version’ for org.apache.hadoop:hadoop-minikdc:jar is missing. @ org.apache.hbase:hbase-server:[unknown-version], /Users/vinishavyasa/work/hbase/hbase-server/pom.xml, line 545, column 17
[ERROR] The project org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT (/Users/vinishavyasa/work/hbase/hbase-spark/pom.xml) has 1 error
[ERROR] ‘dependencies.dependency.version’ for org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar is missing. @ org.apache.hbase:hbase-spark:[unknown-version], /Users/vinishavyasa/work/hbase/hbase-spark/pom.xml, line 498, column 17

Leave a Reply

Your email address will not be published. Required fields are marked *

If you have specific technical questions, please post them in the Forums