Home Forums HDFS set up name node as a hdfs client

This topic contains 9 replies, has 3 voices, and was last updated by  tedr 2 years ago.

  • Creator
    Topic
  • #11927

    Steve Cohen
    Participant

    I set up a two node cluster with the hmc installer and it only set up the secondary box as an hdfs and mapred client. How do I add the primary box to be a client as well?

    Thanks,
    Steve Cohen

Viewing 9 replies - 1 through 9 (of 9 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #12013

    tedr
    Member

    Steve,

    Yup, that’ll work.

    Ted.

    Collapse
    #12012

    Steve Cohen
    Participant

    I found an easier way. Install HDP on one node and then add the second node.

    Collapse
    #12011

    tedr
    Member

    Hi Steve,

    We can’t support modifying the installer code. The process I am working on is manually starting up a datanode and tasktracker on the same box as the namenode, and then hooking it in to the monitoring.

    Ted.

    Collapse
    #12009

    Steve Cohen
    Participant

    Any word on the proper way to configure the datanode on the namenode? I just noticed in the datanode.pp file, there is this if block

    if ($a_namenode_on_node == true){
    $create_pid_dir = false
    $create_log_dir = false
    } else {
    $create_pid_dir = true
    $create_log_dir = true
    }

    Presumably, I would need to change the false to true.

    Collapse
    #11982

    tedr
    Member

    Hi Steve,

    Although you can’t do this through HMC during the install, it is possible to do it manually after HMC has finished. I’m researching and testing the full process and will post it here when I’ve finished.

    Thanks,
    Ted.

    Collapse
    #11977

    Steve Cohen
    Participant

    I understand that it isn’t optimal. On the other hand, I was only given two servers to work with so I had the choice, install VMs on the two servers (which isn’t optimal), install datanodes on both servers (which isn’t optimal), or run only one datanode (which isn’t optimal). Since both boxes have lots of cpus and ram, I went with installing datanodes on both servers.

    Collapse
    #11956

    tedr
    Member

    Hi Steve,

    As Robert put it, this is by design. The HMC installer is set up for production clusters, and on a production cluster it is a bad idea to have a namenode and datanode on the same computer.

    Ted.

    Collapse
    #11946

    Steve Cohen
    Participant

    I went ahead and added it manually by creating the directory, setting the correct ownerships and permissions, and running /usr/lib/hadoop/bin/hadoop-daemon.sh start datanode on the server. I just wonder why isn’t there a way to do it through the web interface.

    Collapse
    #11931

    Robert
    Participant

    Hi Steve,
    I seen this behavior as well and I assume it was designed this way as a best practice for production clusters. I guess you would just have to fire up another node to get another datanode and tasktracker within your cluster.

    HTH
    -Robert

    Collapse
Viewing 9 replies - 1 through 9 (of 9 total)