HDP on Linux – Installation Forum

YARN and MapReduce Memory Configuration Settings on AWS

  • #52877

    Im having lots of troubles to make a Single Node Hadoop work on AWS. Im using an m1.large instance (2 cpu cores, 7,5 GB RAM, 1 disk). I already tried the HDP2.1 and HDP2.0 stacks and with the default configuration the Smoke Tests work fine but then Pig Scripts dont Run. Once I solved a similar problem problem changing the YARN memory settings, but this time I tried again and it doesnt work. Unfortunately I didnt write down the values I used when it did worked,

    Acoording to the YARN Utility Script recomended in:


    I get the following results:

    Using cores=2 memory=7GB disks=1 hbase=True
    Profile: cores=2 memory=4096MB reserved=3GB usableMem=4GB disks=1
    Num Container=3
    Container Ram=1024MB
    Used Ram=3GB
    Unused Ram=3GB

    I did the changes but then Mapreduce or other Smoke tests fail and the jobs still dont run. I also tried several configurations (8 cointainters of 512MB, etc) with no luck.

    Does anybody can suggest a right configuration for YARN and MapReduce memory setting for a single node in AWS m1.large?

    I’ll highly appreciate any help

to create new topics or reply. | New User Registration

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.