how many Childs to run concurrently

to create new topics or reply. | New User Registration

This topic contains 2 replies, has 2 voices, and was last updated by  Sun Ww 1 year, 5 months ago.

  • Creator
  • #47026

    Sun Ww


    my nodemanage machines have 2*6 cores and enable hyper-threading.
    if I run max 18 YarnChilds on each machine concurrently, the cpu usage is often more than 90%. is it too high?

    if I run max 12 YarnChilds on each machine concurrently, the cpu usage is around 50%. Meanwhile I found the “r” column often shows 10~20 by using vmstat.

    which one should I choose, 18 or 12 ?

    Thank you

Viewing 2 replies - 1 through 2 (of 2 total)

You must be to reply to this topic. | Create Account

  • Author
  • #48399

    Sun Ww

    I’ll try it.
    Thank you.


    18 tasks on a 12 core machine is on the edge, but is reasonable. It does look like your workload is cpu intensive, so may be use 15 to hit a balance?

    You may also want to check the number of disks. Running N containers where N > 1.5 * numDisks usually has negative impact on job performance.

Viewing 2 replies - 1 through 2 (of 2 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.