YARN Forum

how many Childs to run concurrently

  • #47026
    Sun Ww


    my nodemanage machines have 2*6 cores and enable hyper-threading.
    if I run max 18 YarnChilds on each machine concurrently, the cpu usage is often more than 90%. is it too high?

    if I run max 12 YarnChilds on each machine concurrently, the cpu usage is around 50%. Meanwhile I found the “r” column often shows 10~20 by using vmstat.

    which one should I choose, 18 or 12 ?

    Thank you

to create new topics or reply. | New User Registration

  • Author
  • #47616

    18 tasks on a 12 core machine is on the edge, but is reasonable. It does look like your workload is cpu intensive, so may be use 15 to hit a balance?

    You may also want to check the number of disks. Running N containers where N > 1.5 * numDisks usually has negative impact on job performance.

    Sun Ww

    I’ll try it.
    Thank you.

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.