HDP on Linux – Installation Forum

How to recover unhealthy node

  • #49510
    Peter Teunissen
    Participant

    I’m installing HDP2.0 on Ubuntu 12.04 as a single node test setup. I got to the point where I’ve got HDFS and YARN installed and running. However the single datanode I’ve got shown as unhealthy in the YARN webapp. The message is 1/1 local-dirs turned bad: /grid/hadoop/hdfs/yarn/local;
    I noticed that I didn’t create the directory. So I created it and updated ownership and access rights. Even after reboots and restarts of the different daemons, the node is still shown as unhealthy.

    What additional things do I need to do to recover the datanode and get it ‘healthy’ again?

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #50440
    Ryan Suarez
    Participant

    Same problem. Anyone?

    #50672
    Dave
    Moderator

    Hi Peter,

    On this datanode can you run ls -al on /grid/hadoop/hdfs/yarn/local and print it here?

    Thanks

    Dave

    #50774
    Peter Teunissen
    Participant

    @Dave:

    root@hadoop:/# ls -al /grid/hadoop/hdfs/yarn/local
    total 8,0K
    drwxr-xr-x 2 hdfs hadoop 4,0K mrt  3 15:47 .
    drwxr-xr-x 3 hdfs hadoop 4,0K mrt  3 15:47 ..
    root@hadoop:/# 
    
    #50778
    Dave
    Moderator

    Hi Peter,

    This directory should be owned by yarn:hadoop
    Can you change this directory and let me know if the problem persists?

    Thanks

    Dave

    #50783
    Peter Teunissen
    Participant

    That solved the issue! After restarting yarn, the node is shown as healthy now.

    thanks

    #73330
    Andrei Bogdan
    Participant

    @Peter Teunissen

    Could you please explain step-by-step what did you do to restart yarn? What commands ?
    I am having an unhealthy node and I cannot use my Hadoop at all. The only solution for me now is to install a new Hadoop distribution.

    Thank you in advance!

    #73333
    Peter Teunissen
    Participant

    @andrei:

    It’s a long time ago and I can’t finde my notes so I can’t give you my step-by-step procedure. But, look at https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html. There you’ll find the yarn commands. I think you need something like

    yarn application -list

    to get the application ID’s, then

    yarn aplication -kill <ID>

    . To start the applications simply do:

    yarn resourcemanager

    ,

    yarn nodemanager

    and

    yarn proxyserver

    .

    HTH

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.