The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

HDP on Linux – Installation Forum

Ruby Dependencies?

  • #7396

    Problems installing with HMC due to Ruby dependencies.

  • Author
  • #7397

    While installing HMC on a 27 node cluster I receive the following error during “Node Discovery and Preparation”:

    Error: Missing Dependency: ruby-augeas is needed by package puppet-2.7.9-2.noarch (HDP-

    same error for ruby-shadow. Has anyone else see this problem?


    BTW, I am using CentOS 5.7.

    Sasha J

    please, make sure you have EPEL repository configured.

    [root@centos57 ~]# yum repolist
    repo id repo name status
    HDP- Hortonworks Data Platform Version – HDP- 110
    base CentOS-5 – Base 3,591
    epel Extra Packages for Enterprise Linux 5 – x86_64 7,070
    extras CentOS-5 – Extras 273
    updates CentOS-5 – Updates 744
    repolist: 11,788
    [root@centos57 ~]#

    Those packages are in EPEL repo:

    [root@centos57 ~]# yum list ruby*

    ruby-augeas.x86_64 0.4.1-1.el5 epel

    ruby-shadow.x86_64 1.4.1-7.el5 epel

    Follow instructions on repository setup here:

    Thank you!


    it looks like the repository is visible on my head node (“admin”):

    [root@admin ~]# yum repolist
    Loaded plugins: fastestmirror, priorities
    Loading mirror speeds from cached hostfile
    * epel:
    24 packages excluded due to repository priority protections
    repo id repo name status
    Centos-5-7 Centos-5-7 3,566
    HDP- Hortonworks Data Platform Version – HDP- 100
    SGI Local SGI Repository 0
    adobe-linux-x86_64 Adobe Systems Incorporated 2
    epel Extra Packages for Enterprise Linux 5 – x86_64 7,046+24
    google-chrome google-chrome – 64-bit 3
    repolist: 10,717

    But, not visible on my compute nodes (n001-n026):

    [root@n001 ~]# yum repolist
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    file:///mnt/iso1/repodata/repomd.xml: [Errno 5] OSError: [Errno 2] No such file or directory: ‘/mnt/iso1/repodata/repomd.xml’
    Trying other mirror.
    repo id repo name status
    Centos-5-7 Centos-5-7 3,566
    HDP- Hortonworks Data Platform Version – HDP- 100
    SGI Local SGI Repository 0
    repolist: 3,666

    There is an error in this listing regarding “file:///mnt/iso1/repodata/repomd.xml”, but it looks like the compute node is successfully reaching the mirror that I set up on the head node.


    Sasha J

    This should be OK.
    Your HMC node suppose to push all needed packages to all other nodes during the first stage of installation.
    And then use your local repo for all HDP packages.
    Please, clean up your compute nodes from that error you mentioned, then please do “yum erase hmc puppet” and then “yum install hmc”. This should clean up you HMC node and install hmc and all the dependencies.

    Thank you!

    Sasha J

    Also, please check if you have those packages already installed.
    There may be versions mismatch.
    Please, remove those 2 packages from your HMC node before installing HMC.
    Then “yum install hmc” will install it asa dependency with the correct version.



    I followed your advice above, cleaning up everything and re-installing HMC, however I continue to get the same ruby missing dependency errors. Looking further in the log file, I see that I’m missing not only ruby-augeas and ruby-shadow, but also ruby and ruby(abi) as well.

    Are there any log files that would be useful to you that I can collect from my machine?


    Sasha J

    No, there is no specific log files during “yum install hmc”…
    Only standard system logs.
    Could you grab the whole output from “yum install hmc” and post it here?
    Also, please, post output of “rpm -qa | grep ruby”

    Thank you!


    here is the output from my HMC install:

    [root@admin brookse]# yum install hmc
    Loaded plugins: fastestmirror, priorities
    Repository HDP- is listed more than once in the configuration
    Loading mirror speeds from cached hostfile
    * epel:
    24 packages excluded due to repository priority protections
    Setting up Install Process
    Resolving Dependencies
    –> Running transaction check
    —> Package hmc.noarch 0:1.0.0-1.el5 set to be updated
    –> Processing Dependency: puppet = 2.7.9 for package: hmc
    –> Processing Dependency: rubygems for package: hmc
    –> Processing Dependency: mod_passenger for package: hmc
    –> Running transaction check
    —> Package mod_passenger.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem-passenger-native-libs = 1:3.0.12-1.el5.centos for package: mod_passenger
    —> Package puppet.noarch 0:2.7.9-2 set to be updated
    –> Processing Dependency: ruby >= 1.8.1 for package: puppet
    –> Processing Dependency: facter >= 1.5 for package: puppet
    –> Processing Dependency: ruby-shadow for package: puppet
    –> Processing Dependency: /usr/bin/ruby for package: puppet
    –> Processing Dependency: ruby-augeas for package: puppet
    —> Package rubygems.noarch 0:1.3.1-1.el5 set to be updated
    –> Processing Dependency: ruby-rdoc for package: rubygems
    –> Running transaction check
    —> Package facter.x86_64 0:1.6.9-1.el5 set to be updated
    —> Package ruby.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    —> Package ruby-augeas.x86_64 0:0.4.1-1.el5 set to be updated
    —> Package ruby-rdoc.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    –> Processing Dependency: ruby-irb = 1.8.5-19.el5_6.1 for package: ruby-rdoc
    —> Package ruby-shadow.x86_64 0:1.4.1-7.el5 set to be updated
    —> Package rubygem-passenger-native-libs.x86_64 1:3.0.12-1.el5.centos_1.8.5 set to be updated
    –> Processing Dependency: rubygem-passenger-native = 1:3.0.12-1.el5.centos for package: rubygem-passenger-native-libs
    –> Running transaction check
    —> Package ruby-irb.x86_64 0:1.8.5-19.el5_6.1 set to be updated
    —> Package rubygem-passenger-native.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem-passenger = 1:3.0.12-1.el5.centos for package: rubygem-passenger-native
    –> Running transaction check
    —> Package rubygem-passenger.x86_64 1:3.0.12-1.el5.centos set to be updated
    –> Processing Dependency: rubygem(fastthread) >= 1.0.1 for package: rubygem-passenger
    –> Processing Dependency: rubygem(daemon_controller) >= 0.2.5 for package: rubygem-passenger
    –> Processing Dependency: rubygem(rake) >= 0.8.1 for package: rubygem-passenger
    –> Processing Dependency: rubygem(rack) for package: rubygem-passenger
    –> Running transaction check
    —> Package rubygem-daemon_controller.noarch 0:0.2.5-1 set to be updated
    —> Package rubygem-fastthread.x86_64 0:1.0.7-1.el5 set to be updated
    —> Package rubygem-rack.noarch 1:1.1.0-2.el5 set to be updated
    —> Package rubygem-rake.noarch 0:0.8.7-2.el5 set to be updated
    –> Finished Dependency Resolution

    Dependencies Resolved

    Package Arch Version Repository Size
    hmc noarch 1.0.0-1.el5 HDP- 14 M
    Installing for dependencies:
    facter x86_64 1.6.9-1.el5 HDP- 69 k
    mod_passenger x86_64 1:3.0.12-1.el5.centos HDP- 1.6 M
    puppet noarch 2.7.9-2 HDP- 1.0 M
    ruby x86_64 1.8.5-19.el5_6.1 Centos-5-7 276 k
    ruby-augeas x86_64 0.4.1-1.el5 epel 21 k
    ruby-irb x86_64 1.8.5-19.el5_6.1 Centos-5-7 69 k
    ruby-rdoc x86_64 1.8.5-19.el5_6.1 Centos-5-7 135 k
    ruby-shadow x86_64 1.4.1-7.el5 epel 9.6 k
    rubygem-daemon_controller noarch 0.2.5-1 HDP- 71 k
    rubygem-fastthread x86_64 1.0.7-1.el5 epel 59 k
    rubygem-passenger x86_64 1:3.0.12-1.el5.centos HDP- 4.6 M
    rubygem-passenger-native x86_64 1:3.0.12-1.el5.centos HDP- 660 k
    rubygem-passenger-native-libs x86_64 1:3.0.12-1.el5.centos_1.8.5 HDP- 34 k
    rubygem-rack noarch 1:1.1.0-2.el5 epel 450 k
    rubygem-rake noarch 0.8.7-2.el5 epel 426 k
    rubygems noarch 1.3.1-1.el5 epel 177 k

    Transaction Summary
    Install 17 Package(s)
    Upgrade 0 Package(s)

    Total download size: 24 M
    Is this ok [y/N]: y
    Downloading Packages:
    (1/17): ruby-shadow-1.4.1-7.el5.x86_64.rpm | 9.6 kB 00:00
    (2/17): ruby-augeas-0.4.1-1.el5.x86_64.rpm | 21 kB 00:00
    (3/17): rubygem-passenger-native-libs-3.0.12-1.el5.centos_1.8.5.x86_64.rpm | 34 kB 00:00
    (4/17): rubygem-fastthread-1.0.7-1.el5.x86_64.rpm | 59 kB 00:01
    (5/17): facter-1.6.9-1.el5.x86_64.rpm | 69 kB 00:00
    (6/17): rubygem-daemon_controller-0.2.5-1.noarch.rpm | 71 kB 00:00
    (7/17): rubygems-1.3.1-1.el5.noarch.rpm | 177 kB 00:05
    (8/17): rubygem-rake-0.8.7-2.el5.noarch.rpm | 426 kB 00:07
    (9/17): rubygem-rack-1.1.0-2.el5.noarch.rpm | 450 kB 00:10
    (10/17): rubygem-passenger-native-3.0.12-1.el5.centos.x86_64.rpm | 660 kB 00:00
    (11/17): puppet-2.7.9-2.noarch.rpm | 1.0 MB 00:00
    (12/17): mod_passenger-3.0.12-1.el5.centos.x86_64.rpm | 1.6 MB 00:00
    (13/17): rubygem-passenger-3.0.12-1.el5.centos.x86_64.rpm | 4.6 MB 00:00
    (14/17): hmc-1.0.0-1.el5.noarch.rpm | 14 MB 00:00
    Total 869 kB/s | 24 MB 00:27
    Running rpm_check_debug
    Running Transaction Test
    Finished Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : ruby 1/17
    Installing : ruby-augeas 2/17
    Installing : ruby-shadow 3/17
    Installing : facter 4/17
    Installing : puppet 5/17
    Installing : ruby-irb 6/17
    Installing : ruby-rdoc 7/17
    Installing : rubygems 8/17
    Installing : rubygem-fastthread 9/17
    Installing : rubygem-rake 10/17
    Installing : rubygem-rack 11/17
    Installing : rubygem-daemon_controller 12/17
    Installing : rubygem-passenger 13/17
    Installing : rubygem-passenger-native 14/17
    Installing : rubygem-passenger-native-libs 15/17
    Installing : mod_passenger 16/17
    Installing : hmc 17/17

    hmc.noarch 0:1.0.0-1.el5

    Dependency Installed:
    facter.x86_64 0:1.6.9-1.el5 mod_passenger.x86_64 1:3.0.12-1.el5.centos
    puppet.noarch 0:2.7.9-2 ruby.x86_64 0:1.8.5-19.el5_6.1
    ruby-augeas.x86_64 0:0.4.1-1.el5 ruby-irb.x86_64 0:1.8.5-19.el5_6.1
    ruby-rdoc.x86_64 0:1.8.5-19.el5_6.1 ruby-shadow.x86_64 0:1.4.1-7.el5
    rubygem-daemon_controller.noarch 0:0.2.5-1 rubygem-fastthread.x86_64 0:1.0.7-1.el5
    rubygem-passenger.x86_64 1:3.0.12-1.el5.centos rubygem-passenger-native.x86_64 1:3.0.12-1.el5.centos
    rubygem-passenger-native-libs.x86_64 1:3.0.12-1.el5.centos_1.8.5 rubygem-rack.noarch 1:1.1.0-2.el5
    rubygem-rake.noarch 0:0.8.7-2.el5 rubygems.noarch 0:1.3.1-1.el5



    …and here is the output from the RPM query:

    [root@admin HDP-]# rpm -qa | grep ruby


    I also find this in the puppet_agent_httpd.log file as I run the “Add Nodes” command. Does this SSL problem look like a possible cause of the Puppet/Ruby dependency problem?

    [2012-07-19 09:34:52] DEBUG Puppet::Network::HTTP::WEBrickREST is mounted on /.
    [2012-07-19 09:34:52] DEBUG #<Puppet::Network::XMLRPC::WEBrickServlet:0x2aaaad6b9cc0 @parser=nil, @class_delim=".", @service_hook=#, @create=nil, @default_handler=nil, @handler=[[“”, #, [[“string”, “string”, “string”]], nil]], @loadedhandlers=[“puppetrunner”], @valid_ip=nil> is mounted on /RPC2.
    [2012-07-19 09:34:52] INFO WEBrick::HTTPServer#start: pid=30385 port=8139
    [2012-07-19 09:35:00] DEBUG accept:
    [2012-07-19 09:35:00] ERROR OpenSSL::SSL::SSLError:
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `call’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:95:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `each’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:23:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:82:in `start’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new’

    Sasha J

    looks like all ruby pieces installed correctly now.
    As of SSL error, I do not think this is a problem, puppet generates and installs new SSL certificates on the first run, those errors are from this stage.

    Let us move to the next step:
    start HMC (“service hmc start”) and go to the installation.

    Thank you!


    I restarted the HMC service, and used the installer, but still the same failure as before. The Ruby dependencies are not getting resolved properly on the compute nodes, even though the head node has the correct Ruby pieces. Is there anything I can look for on the compute nodes to help understand this problem?

    Sasha J

    Any networking issues in there? firewall, SELinux, something else?
    try to run “yum clean all” on all compute nodes, to clean cashed yum data, this way yum will resync repositories on first run.
    make sure all compute nodes can access EPEL repo as well as HDP repo.

    Let us know about the result.

    Thank you!


    Sasha, I successfully got around the ruby-augeas and ruby-shadow dependencies by setting up an epel repository on my mirror (none of my compute nodes have access to the internet), and setting up epel.repo files on each of the compute nodes to point to my mirror.

    cd /var/www/html/rpms
    reposync -r epel
    createrepo /var/www/html/rpms/epel

    Now, however, I have a dependency on ruby to resolve. Ruby and ruby-devel are installed on my head node, but not on any of the compute nodes. I’m looking into how make ruby available on my mirror now.


    Sasha, I solved my Ruby dependency problems this morning (I’m learning way too much about yum repositories!). I was able to get to the “Deployment Progress” screen where there was a failure during the cluster install. The logs indicate “PUPPET_KICK_FAILED”. I’m starting to look into the cause of that now.


    here is the error message from the puppet_apply.log on one of the compute nodes:

    emerson@starfish-2:~$ grep err puppet_apply.log.n001
    Mon Jul 23 13:31:58 -0400 2012 /Stage[1]/Hdp::Snmp/Hdp::Snmp-configfile[snmpd.conf]/Hdp::Configfile[/etc/snmp//snmpd.conf]/File[/etc/snmp//snmpd.conf]/ensure (err): change from absent to present failed: Could not set ‘present on ensure: No such file or directory – /etc/snmp/snmpd.conf.puppettmp_2818 at /etc/puppet/agent/modules/hdp/manifests/configfile.pp:64
    Mon Jul 23 13:32:06 -0400 2012 /Stage[1]/Hdp::Pre_install_pkgs/Hdp::Exec[yum install $pre_installed_pkgs]/Exec[yum install $pre_installed_pkgs]/returns (err): change from notrun to 0 failed: yum install -y hadoop hadoop-libhdfs.x86_64 hadoop-native.x86_64 hadoop-pipes.x86_64 hadoop-sbin.x86_64 hadoop-lzo zookeeper hive hcatalog oozie-client.noarch hdp_mon_nagios_addons nagios-3.2.3 nagios-plugins-1.4.9 fping net-snmp-utils ganglia-gmetad-3.2.0 ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons snappy snappy-devel returned 1 instead of one of [0] at /etc/puppet/agent/modules/hdp/manifests/init.pp:222

    Any thoughts?

    Sasha J

    The reason of the problem is timeout during yum install execution.
    Take a look here:

    Thank you!


    the instructions imply executing a yum install on the head node, but do I also need to pre-install on all of the compute nodes?


    Sasha J

    yes, you better preinstall on all nodes, then rerun HMC installer.

    Thank you!


    Sasha, I continued to have trouble with dependencies using a local mirror, so I opened up access to the internet for all of my compute nodes. I have successfully created a cluster with 10 nodes. Now I want to add additional nodes using HMC. In the page that requests the “Hosts File”, should I provide a file that has just the new node names, or should I make an inclusive list that has all of the existing nodes plus the new nodes?


The forum ‘HDP on Linux – Installation’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.