Home Forums YARN how many Childs to run concurrently

This topic contains 2 replies, has 2 voices, and was last updated by  Sun Ww 7 months, 1 week ago.

  • Creator
    Topic
  • #47026

    Sun Ww
    Participant

    Hi

    my nodemanage machines have 2*6 cores and enable hyper-threading.
    if I run max 18 YarnChilds on each machine concurrently, the cpu usage is often more than 90%. is it too high?

    if I run max 12 YarnChilds on each machine concurrently, the cpu usage is around 50%. Meanwhile I found the “r” column often shows 10~20 by using vmstat.

    which one should I choose, 18 or 12 ?

    Thank you

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #48399

    Sun Ww
    Participant

    I’ll try it.
    Thank you.

    Collapse
    #47616

    18 tasks on a 12 core machine is on the edge, but is reasonable. It does look like your workload is cpu intensive, so may be use 15 to hit a balance?

    You may also want to check the number of disks. Running N containers where N > 1.5 * numDisks usually has negative impact on job performance.

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)