Home Forums YARN yarn-utils.py calculations

This topic contains 0 replies, has 1 voice, and was last updated by  Tom Stewart 4 months, 4 weeks ago.

  • Creator
    Topic
  • #54688

    Tom Stewart
    Participant

    I ran the yarn-utils.py with the following output:

    [scripts]# python yarn-utils.py -c 12 -m 96 -d 6 -k True
    Using cores=12 memory=96GB disks=6 hbase=True
    Profile: cores=12 memory=69632MB reserved=28GB usableMem=68GB disks=6
    Num Container=11
    Container Ram=6144MB
    Used Ram=66GB
    Unused Ram=28GB
    yarn.scheduler.minimum-allocation-mb=6144
    yarn.scheduler.maximum-allocation-mb=67584
    yarn.nodemanager.resource.memory-mb=67584
    mapreduce.map.memory.mb=6144
    mapreduce.map.java.opts=-Xmx4915m
    mapreduce.reduce.memory.mb=6144
    mapreduce.reduce.java.opts=-Xmx4915m
    yarn.app.mapreduce.am.resource.mb=6144
    yarn.app.mapreduce.am.command-opts=-Xmx4915m
    mapreduce.task.io.sort.mb=2457

    When I compare that to the following page:

    http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html

    Shouldn’t these values be equal per the calculation table?

    Configuration File Configuration Setting Value Calculation
    yarn-site.xml yarn.nodemanager.resource.memory-mb = Containers * RAM-per-Container
    yarn-site.xml yarn.scheduler.minimum-allocation-mb = RAM-per-Container
    yarn-site.xml yarn.scheduler.maximum-allocation-mb = containers * RAM-per-Container
    mapred-site.xml mapreduce.map.memory.mb = RAM-per-Container
    mapred-site.xml mapreduce.reduce.memory.mb = 2 * RAM-per-Container
    mapred-site.xml mapreduce.map.java.opts = 0.8 * RAM-per-Container
    mapred-site.xml mapreduce.reduce.java.opts = 0.8 * 2 * RAM-per-Container
    yarn-site.xml (check) yarn.app.mapreduce.am.resource.mb = 2 * RAM-per-Container
    yarn-site.xml (check) yarn.app.mapreduce.am.command-opts = 0.8 * 2 * RAM-per-Container

    The values in the table calculated as “2 * RAM-per-Container” don’t appear to be that way in the python script. What values should I use for my cluster, those I calculate from the web page or take the ones from the script?

You must be logged in to reply to this topic.