The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

HDP on Linux – Installation Forum

How to recover unhealthy node

  • #49510
    Peter Teunissen
    Participant

    I’m installing HDP2.0 on Ubuntu 12.04 as a single node test setup. I got to the point where I’ve got HDFS and YARN installed and running. However the single datanode I’ve got shown as unhealthy in the YARN webapp. The message is 1/1 local-dirs turned bad: /grid/hadoop/hdfs/yarn/local;
    I noticed that I didn’t create the directory. So I created it and updated ownership and access rights. Even after reboots and restarts of the different daemons, the node is still shown as unhealthy.

    What additional things do I need to do to recover the datanode and get it ‘healthy’ again?

  • Author
    Replies
  • #50440
    Ryan Suarez
    Participant

    Same problem. Anyone?

    #50672
    Dave
    Moderator

    Hi Peter,

    On this datanode can you run ls -al on /grid/hadoop/hdfs/yarn/local and print it here?

    Thanks

    Dave

    #50774
    Peter Teunissen
    Participant

    @Dave:

    root@hadoop:/# ls -al /grid/hadoop/hdfs/yarn/local
    total 8,0K
    drwxr-xr-x 2 hdfs hadoop 4,0K mrt  3 15:47 .
    drwxr-xr-x 3 hdfs hadoop 4,0K mrt  3 15:47 ..
    root@hadoop:/# 
    
    #50778
    Dave
    Moderator

    Hi Peter,

    This directory should be owned by yarn:hadoop
    Can you change this directory and let me know if the problem persists?

    Thanks

    Dave

    #50783
    Peter Teunissen
    Participant

    That solved the issue! After restarting yarn, the node is shown as healthy now.

    thanks

    #73330
    Andrei Bogdan
    Participant

    @Peter Teunissen

    Could you please explain step-by-step what did you do to restart yarn? What commands ?
    I am having an unhealthy node and I cannot use my Hadoop at all. The only solution for me now is to install a new Hadoop distribution.

    Thank you in advance!

    #73333
    Peter Teunissen
    Participant

    @andrei:

    It’s a long time ago and I can’t finde my notes so I can’t give you my step-by-step procedure. But, look at https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html. There you’ll find the yarn commands. I think you need something like

    yarn application -list

    to get the application ID’s, then

    yarn aplication -kill <ID>

    . To start the applications simply do:

    yarn resourcemanager

    ,

    yarn nodemanager

    and

    yarn proxyserver

    .

    HTH

The forum ‘HDP on Linux – Installation’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.