HDFS Forum

set up name node as a hdfs client

  • #11927
    Steve Cohen
    Participant

    I set up a two node cluster with the hmc installer and it only set up the secondary box as an hdfs and mapred client. How do I add the primary box to be a client as well?

    Thanks,
    Steve Cohen

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #11931
    Robert
    Participant

    Hi Steve,
    I seen this behavior as well and I assume it was designed this way as a best practice for production clusters. I guess you would just have to fire up another node to get another datanode and tasktracker within your cluster.

    HTH
    -Robert

    #11946
    Steve Cohen
    Participant

    I went ahead and added it manually by creating the directory, setting the correct ownerships and permissions, and running /usr/lib/hadoop/bin/hadoop-daemon.sh start datanode on the server. I just wonder why isn’t there a way to do it through the web interface.

    #11956
    tedr
    Member

    Hi Steve,

    As Robert put it, this is by design. The HMC installer is set up for production clusters, and on a production cluster it is a bad idea to have a namenode and datanode on the same computer.

    Ted.

    #11977
    Steve Cohen
    Participant

    I understand that it isn’t optimal. On the other hand, I was only given two servers to work with so I had the choice, install VMs on the two servers (which isn’t optimal), install datanodes on both servers (which isn’t optimal), or run only one datanode (which isn’t optimal). Since both boxes have lots of cpus and ram, I went with installing datanodes on both servers.

    #11982
    tedr
    Member

    Hi Steve,

    Although you can’t do this through HMC during the install, it is possible to do it manually after HMC has finished. I’m researching and testing the full process and will post it here when I’ve finished.

    Thanks,
    Ted.

    #12009
    Steve Cohen
    Participant

    Any word on the proper way to configure the datanode on the namenode? I just noticed in the datanode.pp file, there is this if block

    if ($a_namenode_on_node == true){
    $create_pid_dir = false
    $create_log_dir = false
    } else {
    $create_pid_dir = true
    $create_log_dir = true
    }

    Presumably, I would need to change the false to true.

    #12011
    tedr
    Member

    Hi Steve,

    We can’t support modifying the installer code. The process I am working on is manually starting up a datanode and tasktracker on the same box as the namenode, and then hooking it in to the monitoring.

    Ted.

    #12012
    Steve Cohen
    Participant

    I found an easier way. Install HDP on one node and then add the second node.

    #12013
    tedr
    Member

    Steve,

    Yup, that’ll work.

    Ted.

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.