HDP 2.0 – YARN API changes

to create new topics or reply. | New User Registration


This topic contains 4 replies, has 3 voices, and was last updated by  Janos Matyas 1 year, 7 months ago.

  • Creator
  • #31271

    Janos Matyas

    Hi All, Ajay

    We are working on a YARN based product and about to release – however the YARN API went through a few changes (new exceptions, package changes, features, etc) between minor versions of 2.x. Due to the nature of the size, frequency and types of jobs we use GPB, not the REST API.

    Currently the HDP 2.0 is coming with the 2.1 version, those who use Spring YARN they have to use 2.0.5, those who are using Continuuity Weave they have to go against 2.0.2 onwards and those who use Storm-YARN they have to use 2.0.3.

    We are an advocate of Hortonwork’s approach to Hadoop and most of the time we recommend our customers to try out our products on HDP – however right now our product is using Spring Data and Storm as well and it can not be run on HDP 2.0.

    My question is – can you let us now whether future releases of HDP 2.0 will use the current API version or you plan to move forward – in Nexus there are already 2.x and 3.x snapshots available for the YARN API.

    We are about to contribute the upgrade(s) back for those who’d like to use Spring Data/Hadoop, however we’d like to know which version(s) to support – and how stable is the current YARN API?


Viewing 4 replies - 1 through 4 (of 4 total)

You must be to reply to this topic. | Create Account

  • Author
  • #32746

    Janos Matyas

    @Arun – Thanks for the clarification.

    It’s not a big issue – good to know that we shall go ahead (smaller changes coming through later in the YARN API are OK) and move all our stack to YARN (after MR we moved HBase with Hoya so we stick with 2.1.1 – for smaller issues around ServiceLauncher will talk with the Hoya guys). We have ‘version’ upgraded HDP2 with Vagrant/Ansible but look forward for the new release when available.



    Arun Murthy

    @Janos – Unfortunately, this is an artifact of the status quo that YARN APIs weren’t fully baked when we released HDP2 Community Preview in June.

    Apache Hadoop 2 is just done stabilizing (hadoop-2.1.0-beta vote is currently in progress) and we will soon release HDP2 Beta which will be stable and compatible with Apache Hadoop 2.x.

    Please let us know if you need any more help. Thanks.



    You should be very accurate with Hadoop libraries. I got some problems with HDP 2.0 VM being incompatible with latest from Apache 2.1.0-SNAPSHOT. And here is my solution, just in case: http://fusionworks.md/connecting-to-yarn-on-hortonworks-hdp-2-0/


    Arun Murthy

    @Janos – The current YARN apis in hadoop-2.1.0-beta (trunk, branch-2 etc.) in ASF are very *close* to being declared ‘stable’. So I encourage you to use any artifacts which are either 2.1.X and above, either in ASF or in the HDP2 Community Preview.


Viewing 4 replies - 1 through 4 (of 4 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.