YARN Forum

HDP 2.0 – YARN API changes

  • #31271
    Janos Matyas
    Participant

    Hi All, Ajay

    We are working on a YARN based product and about to release – however the YARN API went through a few changes (new exceptions, package changes, features, etc) between minor versions of 2.x. Due to the nature of the size, frequency and types of jobs we use GPB, not the REST API.

    Currently the HDP 2.0 is coming with the 2.1 version, those who use Spring YARN they have to use 2.0.5, those who are using Continuuity Weave they have to go against 2.0.2 onwards and those who use Storm-YARN they have to use 2.0.3.

    We are an advocate of Hortonwork’s approach to Hadoop and most of the time we recommend our customers to try out our products on HDP – however right now our product is using Spring Data and Storm as well and it can not be run on HDP 2.0.

    My question is – can you let us now whether future releases of HDP 2.0 will use the current API version or you plan to move forward – in Nexus there are already 2.x and 3.x snapshots available for the YARN API.

    We are about to contribute the upgrade(s) back for those who’d like to use Spring Data/Hadoop, however we’d like to know which version(s) to support – and how stable is the current YARN API?

    Thanks,
    Janos/ITegration

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #32094
    Arun Murthy
    Moderator

    @Janos – The current YARN apis in hadoop-2.1.0-beta (trunk, branch-2 etc.) in ASF are very *close* to being declared ‘stable’. So I encourage you to use any artifacts which are either 2.1.X and above, either in ASF or in the HDP2 Community Preview.

    hth,
    Arun

    #32561
    Member

    You should be very accurate with Hadoop libraries. I got some problems with HDP 2.0 VM being incompatible with latest from Apache 2.1.0-SNAPSHOT. And here is my solution, just in case: http://fusionworks.md/connecting-to-yarn-on-hortonworks-hdp-2-0/

    #32675
    Arun Murthy
    Moderator

    @Janos – Unfortunately, this is an artifact of the status quo that YARN APIs weren’t fully baked when we released HDP2 Community Preview in June.

    Apache Hadoop 2 is just done stabilizing (hadoop-2.1.0-beta vote is currently in progress) and we will soon release HDP2 Beta which will be stable and compatible with Apache Hadoop 2.x.

    Please let us know if you need any more help. Thanks.

    #32746
    Janos Matyas
    Participant

    @Arun – Thanks for the clarification.

    It’s not a big issue – good to know that we shall go ahead (smaller changes coming through later in the YARN API are OK) and move all our stack to YARN (after MR we moved HBase with Hoya so we stick with 2.1.1 – for smaller issues around ServiceLauncher will talk with the Hoya guys). We have ‘version’ upgraded HDP2 with Vagrant/Ansible but look forward for the new release when available.

    Thanks,
    Janos

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.