Home Forums HDP on Windows – Installation MapRed Capacity Scheduler

This topic contains 5 replies, has 4 voices, and was last updated by  tedr 1 year, 4 months ago.

  • Creator
    Topic
  • #19841

    Ted Malone
    Participant

    In playing around with HDP on Windows, I note that the value for the mapred.capacity-scheduler.queue.default.capacity value is set to 50, which effectively limits the job system to 50% of capacity. If I change this value on the TaskTracker host, the service throws an unhandled exception and dies.

    I notice that the default value for this setting on HDP for Linux is 100..

    Any thoughts on this?

    Thanks.

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #21396

    tedr
    Moderator

    Hi Ted,

    Thanks for the info, I do see that there are two queues created there. I am checking into why it was done that way.

    Thanks,
    Ted.

    Collapse
    #21293

    Ted Malone
    Participant

    If you look at the mapred-site.conf file you’ll find that there are 2 queues, one named “default” and the other named “joblauncher”. If you remove the joblauncher queue and then set the capacity of the default queue to 100 in the capacity-scheduler.conf file, everything will work as advertised on a single queue…

    mapred.queue.names
    default,joblauncher
    Comma separated list of queues configured for this jobtracker.

    Collapse
    #21292

    tedr
    Member

    Hi Ted,

    That makes sense, about needing to change both queues to total 100%. I’m looking into why there’d be two queues configured by default on windows in the first place.

    Thanks,
    Ted.

    Collapse
    #21246

    Ted Malone
    Participant

    This is on Windows Server 2012, and the service threw an unhandled exception on starting, no matter what the value was set to other than 50 (i.e 50 works fine, but 51 does not).

    I found the root of the issue. Turns out that HDP on Windows has 2 queues configured by default (as opposed to the single queue by default on HDP Linux) and the configuration must add up to 100% for the capacities. So, if you increase one queue capacity you must reduce the other….

    Collapse
    #21041

    Robert
    Participant

    Hi Ted,
    If you leave the default value of 50 on the tasktracker node, the job runs fine? What value did you change it to and what is the error you received? Also it would be good to know what OS you are using as well.

    Regards,
    Robert

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)