HDP on Linux – Installation Forum

hadoop format formats wrong directory

  • #14082

    Hello,
    when trying to perform a manual installation I have come across a weird behavior when performing the command ‘su – hdfs -c “/usr/lib/hadoop/bin/hadoop namenode -format” ‘.
    Hadoop tries to format a tmp folder (/tmp/hadoop-${user.name}), which is not the folder specified in hdfs-site.xml under the dfs.data.dir / dfs.name.dir.
    As a result, starting the namenode after the format fails since the namenode is not located in the formatted location.

    Investigating this subject shows that the value for the tmp folder is taken from a variable hadoop.tmp.dir, but I can’t find it in hdfs-site.xml or in core-site.xml.
    Am I missing something, how Can I make sure hadoop namenode -format uses the correct variable from hdfs-site.xml?

    Thanks,

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #14089
    Larry Liu
    Moderator

    Hi, Tzur

    Thanks fro trying HDP.

    The property hadoop.tmp.dir is a default property for hadoop. It can be overwriten in core-site.xml. By the way, are you trying to install HDP 1.2?

    Larry

    #14133

    Hello Larry,
    I’m trying to install Hadoop 1.2 manually (RPMs) after the Amabari installation failed.

    I don’t have that property in core-site.xml, but I can add it manually.
    the question is why do I even need to specify it. Why the format option get an invalid data directory and does not use the one specified in hdfs-site.xml in the dfs.data.dir propety

    #14139

    I added the property hadoop.tmp.dir to hdfs-site.xml and re-formatted hdfs. It is still ignoring the property and tries to format:
    /tmp/hadoop-hdfs/dfs/name

    when my data/name dirs are in
    /mnt/hadoop/

    I’m confused :-\

    #14140
    Larry Liu
    Moderator

    Hi, Tzur,

    The hadoop.tmp.dir should be added into core-site.xml. Can you please try again.

    Larry

    #14142
    Larry Liu
    Moderator

    Hi, Tzur,

    Can you clarify if the system is clean when you install manually after ambari installation failed? It is better to start the installation from a clean system.

    Larry

    #14144

    Hi Larry,
    I created new VMs for non-ambari installation. Easier than cleaning a system 😉

    Ok, so I moved the property from hdfs-site.xml to core-site.xml,
    but when trying to format I’m still getting this:
    Re-format filesystem in /tmp/hadoop-hdfs/dfs/name ? (Y or N)

    when the property in core-site.xml is:
    property
    name hadoop.tmp.dir name
    value /mnt/hadoop/dataNode value

    #14150
    Larry Liu
    Moderator

    Hi, Tzur

    Can you please provide the namenode log file and upload according to the following link?

    http://hortonworks.com/community/forums/topic/hmc-installation-support-help-us-help-you/

    Thanks

    Larry

    #14261

    Hi Larry,
    check.sh produces an empty …out file
    There are also errors on /var/db/hmc/data.db which does not exist.

    I appreciate your help, but luckily I am working with some of Hortonworks employees on a mutual project and I’ll try and get their onsite help on this issue.

    Thank you for your time!

    #14322
    Larry Liu
    Moderator

    Hi, Tzur

    The check.sh is for HDP 1.1. I guess it doesn’t work with HDP 1.2.

    Can you please get namenode log only for me?

    Thanks

    Larry

    #14955
    Larry Liu
    Moderator

    Hi, Tzur

    Just want to follow up with you the updated status of your installation.

    Larry

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.