Home Forums YARN HDP 2.0 – YARN API changes

Tagged: 

This topic contains 4 replies, has 3 voices, and was last updated by  Janos Matyas 1 year, 1 month ago.

  • Creator
    Topic
  • #31271

    Janos Matyas
    Participant

    Hi All, Ajay

    We are working on a YARN based product and about to release – however the YARN API went through a few changes (new exceptions, package changes, features, etc) between minor versions of 2.x. Due to the nature of the size, frequency and types of jobs we use GPB, not the REST API.

    Currently the HDP 2.0 is coming with the 2.1 version, those who use Spring YARN they have to use 2.0.5, those who are using Continuuity Weave they have to go against 2.0.2 onwards and those who use Storm-YARN they have to use 2.0.3.

    We are an advocate of Hortonwork’s approach to Hadoop and most of the time we recommend our customers to try out our products on HDP – however right now our product is using Spring Data and Storm as well and it can not be run on HDP 2.0.

    My question is – can you let us now whether future releases of HDP 2.0 will use the current API version or you plan to move forward – in Nexus there are already 2.x and 3.x snapshots available for the YARN API.

    We are about to contribute the upgrade(s) back for those who’d like to use Spring Data/Hadoop, however we’d like to know which version(s) to support – and how stable is the current YARN API?

    Thanks,
    Janos/ITegration

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #32746

    Janos Matyas
    Participant

    @Arun – Thanks for the clarification.

    It’s not a big issue – good to know that we shall go ahead (smaller changes coming through later in the YARN API are OK) and move all our stack to YARN (after MR we moved HBase with Hoya so we stick with 2.1.1 – for smaller issues around ServiceLauncher will talk with the Hoya guys). We have ‘version’ upgraded HDP2 with Vagrant/Ansible but look forward for the new release when available.

    Thanks,
    Janos

    Collapse
    #32675

    Arun Murthy
    Participant

    @Janos – Unfortunately, this is an artifact of the status quo that YARN APIs weren’t fully baked when we released HDP2 Community Preview in June.

    Apache Hadoop 2 is just done stabilizing (hadoop-2.1.0-beta vote is currently in progress) and we will soon release HDP2 Beta which will be stable and compatible with Apache Hadoop 2.x.

    Please let us know if you need any more help. Thanks.

    Collapse
    #32561


    Member

    You should be very accurate with Hadoop libraries. I got some problems with HDP 2.0 VM being incompatible with latest from Apache 2.1.0-SNAPSHOT. And here is my solution, just in case: http://fusionworks.md/connecting-to-yarn-on-hortonworks-hdp-2-0/

    Collapse
    #32094

    Arun Murthy
    Participant

    @Janos – The current YARN apis in hadoop-2.1.0-beta (trunk, branch-2 etc.) in ASF are very *close* to being declared ‘stable’. So I encourage you to use any artifacts which are either 2.1.X and above, either in ASF or in the HDP2 Community Preview.

    hth,
    Arun

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)