Home Forums HDP on Linux – Installation Puppet Kick Failed

This topic contains 11 replies, has 7 voices, and was last updated by  Sasha J 1 year, 8 months ago.

  • Creator
    Topic
  • #6359

    I was wondering if any of you were encountering this issue:

    Here I attempt a puppet kick on my fully qualified domain name and it failed,
    this usually indicates a puppet kick failure in the cluster deployment phase of the
    hmc deployment.

    > puppet kick “FQDN”
    Triggering “FQDN”
    Host “FQDN” failed: Connection refused – connect(2)
    “FQDN” finished with exit code 2
    Failed: “FQDN”

    But I don’t know much about puppet or the way it interacts with SSL certificates and my hostname so any help would be appreciated.

Viewing 11 replies - 1 through 11 (of 11 total)

The topic ‘Puppet Kick Failed’ is closed to new replies.

  • Author
    Replies
  • #12606

    Sasha J
    Moderator

    James,
    thank you for trying HDP2, but this is the wrong thread to post HDP2 questions.
    Please, use relevant thread.

    Thank you!
    Sasha

    Collapse
    #12596

    Chia-Hao Chang
    Participant

    Hi Akki,
    I have try the work around you provided to install HDP 2, but it show me the following message:
    ===========================================================================
    [root@HadoopHDP2 hdp_manual_install_rpm_helper_files-2.0]# yum install -y hadoop hadoop-libhdfs.x86_64 hadoop-native.x86_64 hadoop-pipes.x86_64 hadoop-sbin.x86_64 hadoop-lzo hadoop hadoop-libhdfs.i386 hadoop-native.i386 hadoop-pipes.i386 hadoop-sbin.i386 hadoop-lzo hive hcatalog oozie-client.noarch hdp_mon_dashboard hdp_mon_nagios_addons nagios-3.2.3 nagios-plugins-1.4.9 fping net-snmp-utils ganglia-gmetad-3.2.0 ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons snappy snappy-devel
    Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi, presto, priorities, refresh-packagekit, security, tmprepo,
    : verify, versionlock
    Loading support for CentOS kernel ABI
    Loading mirror speeds from cached hostfile
    * base: centos.ustc.edu.cn
    * epel: mirrors.ustc.edu.cn
    * extras: centos.ustc.edu.cn
    * updates: centos.ustc.edu.cn
    37 packages excluded due to repository priority protections
    Setting up Install Process
    Package hadoop-2.0.2.1-1.el6.x86_64 already installed and latest version
    Package hadoop-libhdfs-2.0.2.1-1.el6.x86_64 already installed and latest version
    No package hadoop-native.x86_64 available.
    No package hadoop-pipes.x86_64 available.
    No package hadoop-sbin.x86_64 available.
    Package hadoop-lzo-0.5.0-1.x86_64 already installed and latest version
    Package hadoop-2.0.2.1-1.el6.x86_64 already installed and latest version
    No package hadoop-libhdfs.i386 available.
    No package hadoop-native.i386 available.
    No package hadoop-pipes.i386 available.
    No package hadoop-sbin.i386 available.
    ===========================================================================

    Could you or someone give me a hand?
    Best Regards
    James Chang

    Collapse
    #7977

    Sasha J
    Moderator

    Akki,
    Is there any particular reason to repeat here exact wording from sticky article?
    “Puppet Failed : Pre Deploy”

    Collapse
    #7974

    Akki Sharma
    Member

    The following simple steps on the HMC master node just solve the issue …

    1. yum erase hmc puppet
    2. yum install hmc
    3. yum install -y hadoop hadoop-libhdfs.x86_64 hadoop-native.x86_64 hadoop-pipes.x86_64 hadoop-sbin.x86_64 hadoop-lzo hadoop hadoop-libhdfs.i386 hadoop-native.i386 hadoop-pipes.i386 hadoop-sbin.i386 hadoop-lzo hive hcatalog oozie-client.noarch hdp_mon_dashboard hdp_mon_nagios_addons nagios-3.2.3 nagios-plugins-1.4.9 fping net-snmp-utils ganglia-gmetad-3.2.0 ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons ganglia-gmond-3.2.0 gweb hdp_mon_ganglia_addons snappy snappy-devel
    4. service hmc start
    5. connect to HMC by browser and run “normal” installation.

    Collapse
    #7732

    Kumar & Che, thanks for the detailed explanation. Maybe this information could be added to the user guide?

    Collapse
    #7497

    che brian
    Member

    Please follow the HMC installation doc and verify this tips as well .
    1.Setup the ssh-shared environment in the cluster nodes [confirm bn nodes ].
    2.Ensure the availability of HDD / RAM size [It is very important ]
    3.Setup the node [On which we are going to run the HMC ] with browser .
    4.Update the FQDN name in all nodes /etc/hosts file.
    5.Verify the puppet ssl sign / requests status between HMC server / other nodes
    6. If not using DNS we can use HMC server name as puppet [Personal opinion / not
    required/ Because normally puppet client ll check with puppet master with the name of puppet]
    7. Download / install epel / hmc repo rpm in HMC node.
    rpm -Uvh http://public-repo-1.hortonworks.com/HDP-1.0.0.12/repos/ centos5/epel-release-5-
    4.noarch.rpm
    rpm -Uvh http://public-repo-1.hortonworks.com/HDP-1.0.0.12/repos/ centos5/hdp-release-
    1.0.0.12-1.el5.noarch.rpm
    8. Install the per-requesties in all nodes as per the HMC doc.
    9. Install hmc in HMC node.
    yum install php-pecl-json hmc
    10. Once installed start the hmc service .and follow the instructions.
    11. If not preferring HMC to download the JDK bin file, Please upload the bin [32,64 jdk ]file in
    /var/run/hmc/downloads folder with appropriate permission settings [chown -R apache:puppet]
    12. Check the Document root settins in apache.
    13. Check the Download permission for jdk files
    [curl -I localhost/downloads/jdk_filename, confirm the 200 response]
    14. Start the httpd service and access the link from HMC node.
    15. http:///hmc/html/index.php.
    16. Follow-up the installation instruction .
    17. At last step calculate the memory to allocate for apps . and make confirm the jdk download
    path + java path settings.
    LOGS :
    1. Verify the /var/log/puppet_agent.log in all nodes getting the proper update or not.
    2. Check in HMC node /var/log/hmc/hmc.log
    3. [puppet kick got failed is not meaning for puppet is not working . It can't completed the installation successfully. We can confirm with verifying the /var/log/yum.log file . if it has installed all packages properly then the problem is resources and java path and mount points ]

    Collapse
    #7455

    Sasha J
    Moderator

    Binish,
    yes, what Kumar say is the good workaround.
    The reason is when you install in AWS from the online repository,”yum install” command may take longer that predefined timeout. IN this case installation failed.
    Timeout is increased in next HMC release, so it should not be an issue when next release will be available.
    But for now, preinstalling all the packages is the easiest workaround and it always work.

    Thank you!
    Sasha

    Collapse
    #7441

    Hi,
    The HMC tries to install, configure and start the services on node using puppet. When services are selected on web interface and are deployed, the master puppet creates a manifest and pushes it on client puppet/puppets and waits for them to complete installations,configure and start required services according to the manifest.
    All works fine but the bottle neck is INSTALLATION over ONLINE repository. as Sasha J said the time out is due to “yum install” running…(may be because low bandwidth or another yum is already running,etc… plz make sure yum-udate is stop) You can reconfirm it from /var/log/puppet_apply.log on client node or on same node incase of single node cluster.
    So, as puppet client nodes does not complete the yum package installation, the timeout error occurs.

    SOLUTION:
    Plz install the packages before you press DEPLOY button on web interface. So that puppet will only ensure the packages are present and configure it according to its requirement and start the services.

    For single node cluster with complete installation of HMC the packages are:
    hadoop
    hadoop-libhdfs.x86_64
    hadoop-native.x86_64
    hadoop-pipes.x86_64
    hadoop-sbin.x86_64
    hadoop-lzo
    hadoop
    hadoop-libhdfs.i386
    hadoop-native.i386
    hadoop-pipes.i386
    hadoop-sbin.i386
    hadoop-lzo

    zookeeper
    hbase
    mysql-server
    hive
    mysql-connector-java-5.0.8-1
    hive
    hcatalog
    oozie.noarch
    extjs-2.2-1
    oozie-client.noarch
    pig.noarch
    sqoop
    mysql-connector-java-5.0.8-1
    templeton
    templeton-tar-pig-0.0.1-1
    templeton-tar-hive-0.0.1-1
    templeton
    hdp_mon_dashboard
    hdp_mon_nagios_addons nagios-3.2.3
    nagios-plugins-1.4.9 fping
    net-snmp-utils
    ganglia-gmetad-3.2.0
    ganglia-gmond-3.2.0
    gweb
    hdp_mon_ganglia_addons
    ganglia-gmond-3.2.0
    gweb
    hdp_mon_ganglia_addons
    snappy
    snappy-devel
    That’s all…
    Once all the packages are installed manually, you can press deploy. Now the time required to configure and start services is less and timeout error wont occur…

    For MultiNode cluster plz install packages according to services distributed on each node.
    Hope That Helps you all…

    Important logs:
    /var/log/hmc/hmc.log
    /var/log/puppet_apply.log (on client node)

    puppet dynamic manifest file:
    /etc/puppet/master/modules/catalog/files/site.pp (this may help in which packages on which node)

    Collapse
    #6464

    Sasha J
    Moderator

    Binish,
    this looks like timeout during the command “yum install” running…
    Please, rebuild your instance from “small” to “large” or “large” and rerun HMC.
    Also, it may be a good idea to reduce amount of components installed on single node cluster, just use HDFS and MapReduce for start.
    But “small” instance is definitely too small to handle single node cluster…

    Collapse
    #6463

    Hi
    I am getting the following error in the hmc.log while using hdp
    I am using amazon instance CentOS release 5.5 (Final) and is a small instance
    Passwordless ssh is working proper
    Trying to start the basic services in a single machine

    2012:06:28 12:04:21][WARN][PuppetInvoker][PuppetInvoker.php:322][waitForResults]: Kick timed out, waited 120 seconds
    [2012:06:28 12:04:22][INFO][PuppetInvoker][PuppetInvoker.php:270][genKickWait]: Kick attempt (3/3)
    [2012:06:28 12:04:24][INFO][PuppetInvoker][PuppetInvoker.php:310][waitForResults]: Waiting for results from ip-10-190-186-35.ec2.internal
    [2012:06:28 12:04:24][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 3-2-0
    [2012:06:28 12:04:29][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 3-2-0

    Array
    (
    [result] => 0
    [error] =>
    [nokick] => Array
    (
    )

    [failed] => Array
    (
    [0] => ip-10-190-186-35.ec2.internal
    )

    [success] => Array
    (
    )

    [timedoutnodes] => Array
    (
    [0] => ip-10-190-186-35.ec2.internal
    )

    )

    [2012:06:28 12:06:30][INFO][Cluster:HDPCluster][Cluster.php:662][_installAllServices]: Persisting puppet report for install HDP
    [2012:06:28 12:06:30][ERROR][Cluster:HDPCluster][Cluster.php:677][_installAllServices]: Puppet kick failed, no successful nodes
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 3-2-0:FAILED: Cluster install:FAILED
    [2012:06:28 12:06:30][INFO][Cluster:HDPCluster][Cluster.php:1039][setState]: HDPCluster – FAILED
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: HDFS – FAILED
    [2012:06:28 12:06:30][INFO][Service: HDFS (HDPCluster)][Service.php:130][setState]: HDFS – FAILED dryRun=
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MAPREDUCE – FAILED
    [2012:06:28 12:06:30][INFO][Service: MAPREDUCE (HDPCluster)][Service.php:130][setState]: MAPREDUCE – FAILED dryRun=
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: DASHBOARD – FAILED
    [2012:06:28 12:06:30][INFO][Service: DASHBOARD (HDPCluster)][Service.php:130][setState]: DASHBOARD – FAILED dryRun=
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: GANGLIA – FAILED
    [2012:06:28 12:06:30][INFO][Service: GANGLIA (HDPCluster)][Service.php:130][setState]: GANGLIA – FAILED dryRun=
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: NAGIOS – FAILED
    [2012:06:28 12:06:30][INFO][Service: NAGIOS (HDPCluster)][Service.php:130][setState]: NAGIOS – FAILED dryRun=
    [2012:06:28 12:06:30][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MISCELLANEOUS – FAILED
    [2012:06:28 12:06:30][INFO][Service: MISCELLANEOUS (HDPCluster)][Service.php:130][setState]: MISCELLANEOUS – FAILED dryRun=
    [2012:06:28 12:06:30][ERROR][Cluster:HDPCluster][Cluster.php:74][_deployHDP]: Failed to install services.
    [2012:06:28 12:06:30][INFO][ClusterMain:TxnId=3][ClusterMain.php:332][]: Completed action=deploy on cluster=HDPCluster, txn=3-0-0, result=-3, error=Puppet kick failed on all nodes
    [2012:06:28 12:06:33][INFO][ClusterState][clusterState.php:19][updateClusterState]: Update Cluster State with {“state”:”DEPLOYMENT_IN_PROGRESS”,”displayName”:”Deployment in progress”,”timeStamp”:1340885193,”context”:{“txnId”:3,”isInPostProcess”:true}}
    [2012:06:28 12:06:33][INFO][ClusterState][clusterState.php:19][updateClusterState]: Update Cluster State with {“state”:”DEPLOYED”,”displayName”:”Deploy failed”,”timeStamp”:1340885193,”context”:{“status”:false,”txnId”:”3″}}
    [2012:06:28 12:06:33][INFO][ClusterState][clusterState.php:19][updateClusterState]: Update Cluster State with {“state”:”DEPLOYED”,”displayName”:”Deploy failed”,”timeStamp”:1340885193,”context”:{“status”:false,”txnId”:”3″,”isInPostProcess”:false,”postProcessSuccessful”:true}}

    Collapse
    #6371

    Sasha J
    Moderator

    Hi Miguel,

    please make sure you have the FQDN of the puppet server resolvable on all machines, including the HMC install machine.

    if you are using the /etc/hosts file then ensure you have the proper host name in the /etc/hosts of all machines in the cluster

    please also verify that the hosts report the correct name, by issuing:

    hostname -f

    from each host, and ensure your hosts entries match

    Thanks for your interest in HDP and let us know if this does not resolve your issue

    Sasha

    Collapse
Viewing 11 replies - 1 through 11 (of 11 total)