Home Forums HDP on Linux – Installation HDP 2.0 Installation on Amazon EC2

This topic contains 7 replies, has 3 voices, and was last updated by  Sushma K 6 months, 3 weeks ago.

  • Creator
    Topic
  • #46719

    RAVI RANJAN
    Participant

    Hello everyone,
    We are planning to install a HDP 2.0 cluster on Linux on Amazon EC2. This is more of a testing phase so everything is on a small scale ie. 2 node cluster. Can anyone share the best available links/docs for the installation. Any prerequisites or do’s and don’ts would be very helpful.

    Thanks
    Ravi

Viewing 7 replies - 1 through 7 (of 7 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #47801

    Sushma K
    Participant

    Hi again,

    I have all my doubts cleared except for the security group rules to be set on ec2 .

    I have 5 instances and so i will have 5 instance ids. Then, in

    CMP
    Port (Service) Source Action
    ALL sg-79c54511 (hadoop) Delete
    TCP
    Port (Service) Source Action
    0 – 65535 sg-79c54511 (hadoop) Delete
    22 (SSH) 0.0.0.0/0 Delete
    80 (HTTP) 0.0.0.0/0 Delete
    7180 0.0.0.0/0 Delete
    8080 (HTTP*) 0.0.0.0/0 Delete
    50000 – 50100 0.0.0.0/0 Delete
    UDP
    Port (Service) Source Action
    0 – 65535 sg-79c54511 (hadoop) Delete

    why is only one instance id shown – sg-79c54511 (hadoop) – what about all the 5 instaces – 1 ambari master, 2 hdp master and 3 hdp slaves ? please help me with this

    Collapse
    #47798

    Sushma K
    Participant

    Hi again,

    I am just following http://hortonworks.com/kb/ambari-on-ec2/ upto the create instances part. And there they increased the root partition to 100GB for all nodes. And my doubt is y so much is needed? Will 10GB on each do just to try out hdp on ec2 ? Not for long run(as of now). Also, if it is long run how much root partition is needed and if it is just to try out how much partition is needed and why ? I read the documentation but I dint get a clear picture.

    Thanks

    Collapse
    #47683

    Sushma K
    Participant

    Hi again,

    I have been comparing http://hortonworks.com/kb/ambari-on-ec2/ with http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.0/bk_using_Ambari_book/content/ambari-chap2.1.2.html for a clear picture.

    So in the main documentation it is given under “Set up the bits” which is the same section as “YUM Install” in HOW TO ON EC2 :

    RHEL/CentOS/Oracle Linux 6.x

    Confirm the repository is configured by checking the repo list.

    yum repolist
    You should see the Ambari and HDP utilities repositories in the list. The version values vary depending the installation.

    repo id repo name
    | AMBARI-1.x | Ambari 1.x
    | HDP-UTILS-1.1.0.16 | Hortonworks Data Platform Utils
    Install the Ambari bits using yum. This also installs PostgreSQL.

    yum install ambari-server

    where as in HOW TO: AMBARI ON EC2 it is :

    cd
    wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/GA/ambari.repo
    cp ambari.repo /etc/yum.repos.d
    yum install epel-release
    yum repolist
    Install and initialize the ambari server:
    yum install ambari-server
    ambari-server setup
    ambari-server start

    Finally , my question is what should be used for HDP2 in place of “wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/GA/ambari.repo and
    cp ambari.repo /etc/yum.repos.d” given in HOWTO : AMBARI ON EC2.

    Thank You.

    Collapse
    #47682

    Sushma K
    Participant

    Hi Dave,
    This is from http://hortonworks.com/kb/ambari-on-ec2/: I am just trying to map the installation for hdp2 from this and the official documentation. I have a doubt here, what is “sg-79c54511″ under source section in security groups. I copy pasted the security groups part here.
    Used the following security group rules:

    ICMP
    Port (Service) Source Action
    ALL sg-79c54511 (hadoop) Delete
    TCP
    Port (Service) Source Action
    0 – 65535 sg-79c54511 (hadoop) Delete
    22 (SSH) 0.0.0.0/0 Delete
    80 (HTTP) 0.0.0.0/0 Delete
    7180 0.0.0.0/0 Delete
    8080 (HTTP*) 0.0.0.0/0 Delete
    50000 – 50100 0.0.0.0/0 Delete
    UDP
    Port (Service) Source Action
    0 – 65535 sg-79c54511 (hadoop) Delete

    Thank you.

    Collapse
    #47638

    Dave
    Moderator

    Hi Sushma,

    If you use http://hortonworks.com/kb/ambari-on-ec2/ then you will see that it is only taking you through what our installation documentation asks you to do. Also note that this link is for HDP 1.3 so you would need to download the correct Ambari repo for HDP 2.0

    The link I provided is the current HDP 2.0 installation documentation. Please read this especially the first chapters so you understand what is required.

    Thanks

    Dave

    Collapse
    #47624

    Sushma K
    Participant

    Hi Dave,
    What about this ” http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.2/bk_using_Ambari_book/content/ambari-chap1.html ” provided on ” http://hortonworks.com/kb/ambari-on-ec2/ “or the link you provided ?. I am quite new and quite confused. Thank you.

    Collapse
    #46734

    Dave
    Moderator

    Hi Ravi,

    If you’re going to use Ambari then I would suggest using the ambari installation documentation found here:

    http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.0/bk_using_Ambari_book/content/ambari-chap1-1.html

    Thanks

    Dave

    Collapse
Viewing 7 replies - 1 through 7 (of 7 total)