Home Forums HDP on Linux – Installation Ruby Dependencies?

Tagged: 

This topic contains 21 replies, has 2 voices, and was last updated by  Emerson Brooks 1 year, 11 months ago.

  • Creator
    Topic
  • #7396

    Problems installing with HMC due to Ruby dependencies.

Viewing 21 replies - 1 through 21 (of 21 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #7944

    Sasha, I continued to have trouble with dependencies using a local mirror, so I opened up access to the internet for all of my compute nodes. I have successfully created a cluster with 10 nodes. Now I want to add additional nodes using HMC. In the page that requests the “Hosts File”, should I provide a file that has just the new node names, or should I make an inclusive list that has all of the existing nodes plus the new nodes?

    Emerson

    Collapse
    #7593

    Sasha J
    Moderator

    Emerson,
    yes, you better preinstall on all nodes, then rerun HMC installer.

    Thank you!
    Sasha

    Collapse
    #7592

    Sasha,
    the instructions imply executing a yum install on the head node, but do I also need to pre-install on all of the compute nodes?

    Emerson

    Collapse
    #7579

    Sasha J
    Moderator

    The reason of the problem is timeout during yum install execution.
    Take a look here:

    http://hortonworks.com/community/forums/topic/puppet-failed-no-cert/

    Thank you!
    Sasha

    Collapse
    #7577

    Sasha,
    here is the error message from the puppet_apply.log on one of the compute nodes:

    emerson@starfish-2:~$ grep err puppet_apply.log.n001
    Mon Jul 23 13:31:58 -0400 2012 /Stage[1]/Hdp::Snmp/Hdp::Snmp-configfile[snmpd.conf]/Hdp::Configfile[/etc/snmp//snmpd.conf]/File[/etc/snmp//snmpd.conf]/ensure (err): change from absent to present failed: Could not set ‘present on ensure: No such file or directory – /etc/snmp/snmpd.conf.puppettmp_2818 at /etc/puppet/agent/modules/hdp/manifests/configfile.pp:64
    Mon Jul 23 13:32:06 -0400 2012 /Stage[1]/Hdp::Pre_install_pkgs/Hdp::Exec[yum install $pre_installed_pkgs]/Exec[yum install $pre_installed_pkgs]/returns (err): change from notrun to 0 failed: yum install -y hadoop hadoop-libhdfs.x86_64 hadoop-native.x86_64 hadoop-pipes.x86_64 hadoop-sbin.x86_64 hadoop-lzo zookeeper hive hcatalog oozie-client.noarch hdp_mon_nagios_addons nagios-3.2.3 nagios-plugins-1.4.9 fping net-snmp-utils ganglia-gmetad-3.2.0 ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons snappy snappy-devel returned 1 instead of one of [0] at /etc/puppet/agent/modules/hdp/manifests/init.pp:222

    Any thoughts?

    Collapse
    #7562

    Sasha, I solved my Ruby dependency problems this morning (I’m learning way too much about yum repositories!). I was able to get to the “Deployment Progress” screen where there was a failure during the cluster install. The logs indicate “PUPPET_KICK_FAILED”. I’m starting to look into the cause of that now.

    Collapse
    #7561

    Sasha, I successfully got around the ruby-augeas and ruby-shadow dependencies by setting up an epel repository on my mirror (none of my compute nodes have access to the internet), and setting up epel.repo files on each of the compute nodes to point to my mirror.

    cd /var/www/html/rpms
    reposync -r epel
    createrepo /var/www/html/rpms/epel

    Now, however, I have a dependency on ruby to resolve. Ruby and ruby-devel are installed on my head node, but not on any of the compute nodes. I’m looking into how make ruby available on my mirror now.

    Collapse
    #7475

    Sasha J
    Moderator

    Any networking issues in there? firewall, SELinux, something else?
    try to run “yum clean all” on all compute nodes, to clean cashed yum data, this way yum will resync repositories on first run.
    make sure all compute nodes can access EPEL repo as well as HDP repo.

    Let us know about the result.

    Thank you!
    Sasha

    Collapse
    #7464

    Sasha,
    I restarted the HMC service, and used the installer, but still the same failure as before. The Ruby dependencies are not getting resolved properly on the compute nodes, even though the head node has the correct Ruby pieces. Is there anything I can look for on the compute nodes to help understand this problem?

    Collapse
    #7446

    Sasha J
    Moderator

    OK,
    looks like all ruby pieces installed correctly now.
    As of SSL error, I do not think this is a problem, puppet generates and installs new SSL certificates on the first run, those errors are from this stage.

    Let us move to the next step:
    start HMC (“service hmc start”) and go to the installation.

    Thank you!
    Sasha

    Collapse
    #7445

    Sasha,
    I also find this in the puppet_agent_httpd.log file as I run the “Add Nodes” command. Does this SSL problem look like a possible cause of the Puppet/Ruby dependency problem?

    [2012-07-19 09:34:52] DEBUG Puppet::Network::HTTP::WEBrickREST is mounted on /.
    [2012-07-19 09:34:52] DEBUG #<Puppet::Network::XMLRPC::WEBrickServlet:0x2aaaad6b9cc0 @parser=nil, @class_delim=".", @service_hook=#, @create=nil, @default_handler=nil, @handler=[["puppetrunner.run", #, [["string", "string", "string"]], nil]], @loadedhandlers=["puppetrunner"], @valid_ip=nil> is mounted on /RPC2.
    [2012-07-19 09:34:52] INFO WEBrick::HTTPServer#start: pid=30385 port=8139
    [2012-07-19 09:35:00] DEBUG accept: 10.10.32.31:50343
    [2012-07-19 09:35:00] ERROR OpenSSL::SSL::SSLError:
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `call’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:95:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `each’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:23:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:82:in `start’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:

    Collapse
    #7443

    …and here is the output from the RPM query:

    [root@admin HDP-1.0.0.12]# rpm -qa | grep ruby
    ruby-libs-1.8.5-19.el5_6.1
    ruby-devel-1.8.5-19.el5_6.1
    ruby-augeas-0.4.1-1.el5
    rubygem-rake-0.8.7-2.el5
    ruby-devel-1.8.5-19.el5_6.1
    ruby-rdoc-1.8.5-19.el5_6.1
    rubygem-passenger-native-libs-3.0.12-1.el5.centos_1.8.5
    ruby-libs-1.8.5-19.el5_6.1
    rubygem-daemon_controller-0.2.5-1
    ruby-1.8.5-19.el5_6.1
    rubygem-fastthread-1.0.7-1.el5
    rubygem-passenger-3.0.12-1.el5.centos
    ruby-irb-1.8.5-19.el5_6.1
    rubygem-passenger-native-3.0.12-1.el5.centos
    ruby-shadow-1.4.1-7.el5
    rubygem-rack-1.1.0-2.el5
    rubygems-1.3.1-1.el5

    Collapse
    #7442

    Sasha,
    here is the output from my HMC install:

    [root@admin brookse]# yum install hmc
    Loaded plugins: fastestmirror, priorities
    Repository HDP-1.0.0.12 is listed more than once in the configuration
    Loading mirror speeds from cached hostfile
    * epel: mirror.cogentco.com
    24 packages excluded due to repository priority protections
    Setting up Install Process
    Resolving Dependencies
    –> Running transaction check
    —> Package hmc.noarch 0:1.0.0-1.el5 set to be updated
    –> Processing Dependency: puppet = 2.7.9 for package: hmc
    –> Processing Dependency: rubygems for package: hmc
    –> Processing Dependency: mod_passenger for package: hmc
    –> Running transaction check
    —> Package mod_passenger.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem-passenger-native-libs = 1:3.0.12-1.el5.centos for package: mod_passenger
    —> Package puppet.noarch 0:2.7.9-2 set to be updated
    –> Processing Dependency: ruby >= 1.8.1 for package: puppet
    –> Processing Dependency: facter >= 1.5 for package: puppet
    –> Processing Dependency: ruby-shadow for package: puppet
    –> Processing Dependency: /usr/bin/ruby for package: puppet
    –> Processing Dependency: ruby-augeas for package: puppet
    —> Package rubygems.noarch 0:1.3.1-1.el5 set to be updated
    –> Processing Dependency: ruby-rdoc for package: rubygems
    –> Running transaction check
    —> Package facter.x86_64 0:1.6.9-1.el5 set to be updated
    —> Package ruby.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    —> Package ruby-augeas.x86_64 0:0.4.1-1.el5 set to be updated
    —> Package ruby-rdoc.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    –> Processing Dependency: ruby-irb = 1.8.5-19.el5_6.1 for package: ruby-rdoc
    —> Package ruby-shadow.x86_64 0:1.4.1-7.el5 set to be updated
    —> Package rubygem-passenger-native-libs.x86_64 1:3.0.12-1.el5.centos_1.8.5 set to be updated
    –> Processing Dependency: rubygem-passenger-native = 1:3.0.12-1.el5.centos for package: rubygem-passenger-native-libs
    –> Running transaction check
    —> Package ruby-irb.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    —> Package rubygem-passenger-native.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem-passenger = 1:3.0.12-1.el5.centos for package: rubygem-passenger-native
    –> Running transaction check
    —> Package rubygem-passenger.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem(fastthread) >= 1.0.1 for package: rubygem-passenger
    –> Processing Dependency: rubygem(daemon_controller) >= 0.2.5 for package: rubygem-passenger
    –> Processing Dependency: rubygem(rake) >= 0.8.1 for package: rubygem-passenger
    –> Processing Dependency: rubygem(rack) for package: rubygem-passenger
    –> Running transaction check
    —> Package rubygem-daemon_controller.noarch 0:0.2.5-1 set to be updated
    —> Package rubygem-fastthread.x86_64 0:1.0.7-1.el5 set to be updated
    —> Package rubygem-rack.noarch 1:1.1.0-2.el5 set to be updated
    —> Package rubygem-rake.noarch 0:0.8.7-2.el5 set to be updated
    –> Finished Dependency Resolution

    Dependencies Resolved

    ======================================================================================================================================
    Package Arch Version Repository Size
    ======================================================================================================================================
    Installing:
    hmc noarch 1.0.0-1.el5 HDP-1.0.0.12 14 M
    Installing for dependencies:
    facter x86_64 1.6.9-1.el5 HDP-1.0.0.12 69 k
    mod_passenger x86_64 1:3.0.12-1.el5.centos HDP-1.0.0.12 1.6 M
    puppet noarch 2.7.9-2 HDP-1.0.0.12 1.0 M
    ruby x86_64 1.8.5-19.el5_6.1 Centos-5-7 276 k
    ruby-augeas x86_64 0.4.1-1.el5 epel 21 k
    ruby-irb x86_64 1.8.5-19.el5_6.1 Centos-5-7 69 k
    ruby-rdoc x86_64 1.8.5-19.el5_6.1 Centos-5-7 135 k
    ruby-shadow x86_64 1.4.1-7.el5 epel 9.6 k
    rubygem-daemon_controller noarch 0.2.5-1 HDP-1.0.0.12 71 k
    rubygem-fastthread x86_64 1.0.7-1.el5 epel 59 k
    rubygem-passenger x86_64 1:3.0.12-1.el5.centos HDP-1.0.0.12 4.6 M
    rubygem-passenger-native x86_64 1:3.0.12-1.el5.centos HDP-1.0.0.12 660 k
    rubygem-passenger-native-libs x86_64 1:3.0.12-1.el5.centos_1.8.5 HDP-1.0.0.12 34 k
    rubygem-rack noarch 1:1.1.0-2.el5 epel 450 k
    rubygem-rake noarch 0.8.7-2.el5 epel 426 k
    rubygems noarch 1.3.1-1.el5 epel 177 k

    Transaction Summary
    ======================================================================================================================================
    Install 17 Package(s)
    Upgrade 0 Package(s)

    Total download size: 24 M
    Is this ok [y/N]: y
    Downloading Packages:
    (1/17): ruby-shadow-1.4.1-7.el5.x86_64.rpm | 9.6 kB 00:00
    (2/17): ruby-augeas-0.4.1-1.el5.x86_64.rpm | 21 kB 00:00
    (3/17): rubygem-passenger-native-libs-3.0.12-1.el5.centos_1.8.5.x86_64.rpm | 34 kB 00:00
    (4/17): rubygem-fastthread-1.0.7-1.el5.x86_64.rpm | 59 kB 00:01
    (5/17): facter-1.6.9-1.el5.x86_64.rpm | 69 kB 00:00
    (6/17): rubygem-daemon_controller-0.2.5-1.noarch.rpm | 71 kB 00:00
    (7/17): rubygems-1.3.1-1.el5.noarch.rpm | 177 kB 00:05
    (8/17): rubygem-rake-0.8.7-2.el5.noarch.rpm | 426 kB 00:07
    (9/17): rubygem-rack-1.1.0-2.el5.noarch.rpm | 450 kB 00:10
    (10/17): rubygem-passenger-native-3.0.12-1.el5.centos.x86_64.rpm | 660 kB 00:00
    (11/17): puppet-2.7.9-2.noarch.rpm | 1.0 MB 00:00
    (12/17): mod_passenger-3.0.12-1.el5.centos.x86_64.rpm | 1.6 MB 00:00
    (13/17): rubygem-passenger-3.0.12-1.el5.centos.x86_64.rpm | 4.6 MB 00:00
    (14/17): hmc-1.0.0-1.el5.noarch.rpm | 14 MB 00:00
    ————————————————————————————————————————————–
    Total 869 kB/s | 24 MB 00:27
    Running rpm_check_debug
    Running Transaction Test
    Finished Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : ruby 1/17
    Installing : ruby-augeas 2/17
    Installing : ruby-shadow 3/17
    Installing : facter 4/17
    Installing : puppet 5/17
    Installing : ruby-irb 6/17
    Installing : ruby-rdoc 7/17
    Installing : rubygems 8/17
    Installing : rubygem-fastthread 9/17
    Installing : rubygem-rake 10/17
    Installing : rubygem-rack 11/17
    Installing : rubygem-daemon_controller 12/17
    Installing : rubygem-passenger 13/17
    Installing : rubygem-passenger-native 14/17
    Installing : rubygem-passenger-native-libs 15/17
    Installing : mod_passenger 16/17
    Installing : hmc 17/17

    Installed:
    hmc.noarch 0:1.0.0-1.el5

    Dependency Installed:
    facter.x86_64 0:1.6.9-1.el5 mod_passenger.x86_64 1:3.0.12-1.el5.centos
    puppet.noarch 0:2.7.9-2 ruby.x86_64 0:1.8.5-19.el5_6.1
    ruby-augeas.x86_64 0:0.4.1-1.el5 ruby-irb.x86_64 0:1.8.5-19.el5_6.1
    ruby-rdoc.x86_64 0:1.8.5-19.el5_6.1 ruby-shadow.x86_64 0:1.4.1-7.el5
    rubygem-daemon_controller.noarch 0:0.2.5-1 rubygem-fastthread.x86_64 0:1.0.7-1.el5
    rubygem-passenger.x86_64 1:3.0.12-1.el5.centos rubygem-passenger-native.x86_64 1:3.0.12-1.el5.centos
    rubygem-passenger-native-libs.x86_64 1:3.0.12-1.el5.centos_1.8.5 rubygem-rack.noarch 1:1.1.0-2.el5
    rubygem-rake.noarch 0:0.8.7-2.el5 rubygems.noarch 0:1.3.1-1.el5

    Complete!

    Collapse
    #7409

    Sasha J
    Moderator

    No, there is no specific log files during “yum install hmc”…
    Only standard system logs.
    Could you grab the whole output from “yum install hmc” and post it here?
    Also, please, post output of “rpm -qa | grep ruby”

    Thank you!
    Sasha

    Collapse
    #7408

    Sasha,
    I followed your advice above, cleaning up everything and re-installing HMC, however I continue to get the same ruby missing dependency errors. Looking further in the log file, I see that I’m missing not only ruby-augeas and ruby-shadow, but also ruby and ruby(abi) as well.

    Are there any log files that would be useful to you that I can collect from my machine?

    Emerson

    Collapse
    #7407

    Sasha J
    Moderator

    Also, please check if you have those packages already installed.
    There may be versions mismatch.
    Please, remove those 2 packages from your HMC node before installing HMC.
    Then “yum install hmc” will install it asa dependency with the correct version.

    Sasha

    Collapse
    #7406

    Sasha J
    Moderator

    This should be OK.
    Your HMC node suppose to push all needed packages to all other nodes during the first stage of installation.
    And then use your local repo for all HDP packages.
    Please, clean up your compute nodes from that error you mentioned, then please do “yum erase hmc puppet” and then “yum install hmc”. This should clean up you HMC node and install hmc and all the dependencies.

    Thank you!
    Sasha

    Collapse
    #7405

    Sash,
    it looks like the repository is visible on my head node (“admin”):

    [root@admin ~]# yum repolist
    Loaded plugins: fastestmirror, priorities
    Loading mirror speeds from cached hostfile
    * epel: mirror.itc.virginia.edu
    24 packages excluded due to repository priority protections
    repo id repo name status
    Centos-5-7 Centos-5-7 3,566
    HDP-1.0.0.12 Hortonworks Data Platform Version – HDP-1.0.0.12 100
    SGI Local SGI Repository 0
    adobe-linux-x86_64 Adobe Systems Incorporated 2
    epel Extra Packages for Enterprise Linux 5 – x86_64 7,046+24
    google-chrome google-chrome – 64-bit 3
    repolist: 10,717

    But, not visible on my compute nodes (n001-n026):

    [root@n001 ~]# yum repolist
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    file:///mnt/iso1/repodata/repomd.xml: [Errno 5] OSError: [Errno 2] No such file or directory: ‘/mnt/iso1/repodata/repomd.xml’
    Trying other mirror.
    repo id repo name status
    Centos-5-7 Centos-5-7 3,566
    HDP-1.0.0.12 Hortonworks Data Platform Version – HDP-1.0.0.12 100
    SGI Local SGI Repository 0
    repolist: 3,666

    There is an error in this listing regarding “file:///mnt/iso1/repodata/repomd.xml”, but it looks like the compute node is successfully reaching the mirror that I set up on the head node.

    Emerson

    Collapse
    #7403

    Sasha J
    Moderator

    Emerson,
    please, make sure you have EPEL repository configured.

    [root@centos57 ~]# yum repolist
    repo id repo name status
    HDP-1.0.0.12 Hortonworks Data Platform Version – HDP-1.0.0.12 110
    base CentOS-5 – Base 3,591
    epel Extra Packages for Enterprise Linux 5 – x86_64 7,070
    extras CentOS-5 – Extras 273
    updates CentOS-5 – Updates 744
    repolist: 11,788
    [root@centos57 ~]#

    Those packages are in EPEL repo:

    [root@centos57 ~]# yum list ruby*

    ruby-augeas.x86_64 0.4.1-1.el5 epel

    ruby-shadow.x86_64 1.4.1-7.el5 epel

    Follow instructions on repository setup here:

    http://hortonworks.com/download/thankyou_hdp1a/

    Thank you!
    Sasha

    Collapse
    #7398

    BTW, I am using CentOS 5.7.

    Collapse
    #7397

    While installing HMC on a 27 node cluster I receive the following error during “Node Discovery and Preparation”:

    Error: Missing Dependency: ruby-augeas is needed by package puppet-2.7.9-2.noarch (HDP-1.0.0.12)

    same error for ruby-shadow. Has anyone else see this problem?

    Collapse
Viewing 21 replies - 1 through 21 (of 21 total)