Home Forums Ambari dfs.name.dir got cleaned up after power recycle

This topic contains 5 replies, has 2 voices, and was last updated by  tedr 1 year, 2 months ago.

  • Creator
    Topic
  • #27436

    Lohit Giri
    Participant

    Hi,

    i had my 3 nodes cluster running, using ambari 1.2 i was able to configure all the services. unfotunately there was a power reset happened on all nodes.

    i brought up ambari-agents and ambari server and try to restart the cluster, namenode logs says as below -

    2013-06-13 10:48:09,184 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
    java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:412)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:380)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:287)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:548)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1431)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1440)
    2013-06-13 10:48:09,185 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:412)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:380)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:287)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:548)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1431)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1440)

    2013-06-13 10:48:09,186 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

    i tried to look at the dfs.name.dir path and there is nothing there, same thing on the secondary namenode, the checkpoint dir had no fsimage. can ambari trigger this behavior ? i ‘m still looking into this issue, would anyone have idea. why the dfs.name.dir and checkpoint.dir get cleaned up?

    thanks,

Viewing 5 replies - 1 through 5 (of 5 total)

The topic ‘dfs.name.dir got cleaned up after power recycle’ is closed to new replies.

  • Author
    Replies
  • #27527

    tedr
    Moderator

    Hi Lohit,

    Thanks for letting us know you figured it out.

    Ted.

    Collapse
    #27525

    Lohit Giri
    Participant

    Hi Ted,

    thanks for helping me here. i figure out the problem, it was wrong fstab file. the mounts were not mounted using labelled rather than /dev/sd*. It works great now. Thanks a lot for your responses, really appreciate it!

    Regards,
    Lohit

    Collapse
    #27524

    tedr
    Moderator

    Hi Lohit,

    Ambari is not supposed to remove these files. Though usually while running the namenode data is held in RAM and may not have been written to the disk at the time of the outage, though why the secondarynamenode data is missing still puzzles me.

    Thanks,
    Ted.

    Collapse
    #27471

    Lohit Giri
    Participant

    Thanks for reponse Ted.

    Yes i checked that one too. postgres was running. i was able to see all my configs and hosts, agents were up too.

    could you please help me understand, does ambari server removes the fsimage and keep it in db somewhere?

    thanks,
    lg

    Collapse
    #27463

    tedr
    Moderator

    Hi Lohit,

    This sounds like the Postgres database did not come back online. You can check to see if it is running by executing "ps -eaf|grep postgres’ if that gives you nothing but the line showing your grep then you can fire up postgres by executing the following command: "service postgresql start"

    Thanks,
    Ted.

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)