Home Forums HDP on Linux – Installation hadoop format formats wrong directory

This topic contains 10 replies, has 2 voices, and was last updated by  Larry Liu 1 year, 8 months ago.

  • Creator
    Topic
  • #14082

    Hello,
    when trying to perform a manual installation I have come across a weird behavior when performing the command ‘su – hdfs -c “/usr/lib/hadoop/bin/hadoop namenode -format” ‘.
    Hadoop tries to format a tmp folder (/tmp/hadoop-${user.name}), which is not the folder specified in hdfs-site.xml under the dfs.data.dir / dfs.name.dir.
    As a result, starting the namenode after the format fails since the namenode is not located in the formatted location.

    Investigating this subject shows that the value for the tmp folder is taken from a variable hadoop.tmp.dir, but I can’t find it in hdfs-site.xml or in core-site.xml.
    Am I missing something, how Can I make sure hadoop namenode -format uses the correct variable from hdfs-site.xml?

    Thanks,

Viewing 10 replies - 1 through 10 (of 10 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #14955

    Larry Liu
    Moderator

    Hi, Tzur

    Just want to follow up with you the updated status of your installation.

    Larry

    Collapse
    #14322

    Larry Liu
    Moderator

    Hi, Tzur

    The check.sh is for HDP 1.1. I guess it doesn’t work with HDP 1.2.

    Can you please get namenode log only for me?

    Thanks

    Larry

    Collapse
    #14261

    Hi Larry,
    check.sh produces an empty …out file
    There are also errors on /var/db/hmc/data.db which does not exist.

    I appreciate your help, but luckily I am working with some of Hortonworks employees on a mutual project and I’ll try and get their onsite help on this issue.

    Thank you for your time!

    Collapse
    #14150

    Larry Liu
    Moderator

    Hi, Tzur

    Can you please provide the namenode log file and upload according to the following link?

    http://hortonworks.com/community/forums/topic/hmc-installation-support-help-us-help-you/

    Thanks

    Larry

    Collapse
    #14144

    Hi Larry,
    I created new VMs for non-ambari installation. Easier than cleaning a system ;)

    Ok, so I moved the property from hdfs-site.xml to core-site.xml,
    but when trying to format I’m still getting this:
    Re-format filesystem in /tmp/hadoop-hdfs/dfs/name ? (Y or N)

    when the property in core-site.xml is:
    property
    name hadoop.tmp.dir name
    value /mnt/hadoop/dataNode value

    Collapse
    #14142

    Larry Liu
    Moderator

    Hi, Tzur,

    Can you clarify if the system is clean when you install manually after ambari installation failed? It is better to start the installation from a clean system.

    Larry

    Collapse
    #14140

    Larry Liu
    Moderator

    Hi, Tzur,

    The hadoop.tmp.dir should be added into core-site.xml. Can you please try again.

    Larry

    Collapse
    #14139

    I added the property hadoop.tmp.dir to hdfs-site.xml and re-formatted hdfs. It is still ignoring the property and tries to format:
    /tmp/hadoop-hdfs/dfs/name

    when my data/name dirs are in
    /mnt/hadoop/

    I’m confused :-\

    Collapse
    #14133

    Hello Larry,
    I’m trying to install Hadoop 1.2 manually (RPMs) after the Amabari installation failed.

    I don’t have that property in core-site.xml, but I can add it manually.
    the question is why do I even need to specify it. Why the format option get an invalid data directory and does not use the one specified in hdfs-site.xml in the dfs.data.dir propety

    Collapse
    #14089

    Larry Liu
    Moderator

    Hi, Tzur

    Thanks fro trying HDP.

    The property hadoop.tmp.dir is a default property for hadoop. It can be overwriten in core-site.xml. By the way, are you trying to install HDP 1.2?

    Larry

    Collapse
Viewing 10 replies - 1 through 10 (of 10 total)