Home Forums HDP on Linux – Installation 'command not found' HMC script error and stuck HDP install

This topic contains 11 replies, has 3 voices, and was last updated by  Annie Wong 2 years, 1 month ago.

  • Creator
    Topic
  • #6142

    Annie Wong
    Member

    I’m running CentOS 5.7. I tried using HMC to install HDP, but ran into the following error after providing my ssh key and host file:

    /var/run/hmc/clusters/dev/findSshableNodes//ssh.sh: line 1: ssh: command not found

    To workaround this error, I changed /usr/share/hmc/php/frontend/commandUtils.php to include the full path of ‘ssh’ on my box, ie:

    53c53
    < $sshCmd = “ssh -o ConnectTimeOut=3 -o StrictHostKeyChecking=no”;

    > $sshCmd = “/usr/local/bin/ssh -o ConnectTimeOut=3 -o StrictHostKeyChecking=no”;

    This got me further, but I ran into a couple similar errors, and also had to provide the full path to ‘scp’ and ‘service’:

    /usr/share/hmc/php/frontend/addNodes/bootstrap.php workaround:

    29c29
    < $scpCmd = “/usr/local/bin/scp -o StrictHostKeyChecking=no “;

    > $scpCmd = “scp -o StrictHostKeyChecking=no “;

    /usr/share/hmc/ShellScripts/puppet_agent_install.sh workaround:

    205c205
    < out=`service hmc-agent stop`

    > out=`/sbin/service hmc-agent stop`
    209c209
    < out=`service hmc-agent start`

    > out=`/sbin/service hmc-agent start`

    Seems like $PATH isn’t set properly. Are the above commands being run using the ‘puppet’ userid? My /etc/passwd file has:

    puppet:x:52:52:Puppet:/var/lib/puppet:/sbin/nologin

    I temporarily changed puppet’s shell to /usr/local/bin/bash, logged in as puppet and confirmed $PATH, undid my changes in /usr/share/hmc/, and tried HMC again, but I ended up getting the same errors described above.

    With the above workarounds, I’m able to get to the screen in HMC which allows me to install HDP, but the install itself gets stuck and hmc.log continuously prints:

    [2012:06:21 15:40:00][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0

    Full hmc.log below.

    Not sure if it’s stuck because of the broken $PATH issue described above (cause maybe there are other scripts I need to hack to specify full executable paths) or some other reason. Any ideas?

    Thx,
    -Annie

    hmc.log:

    [2012:06:21 15:35:36][INFO][HMC][HMC.php:476][internalTrigger]: HMC triggering action=deployHDP, Deploying cluster
    [2012:06:21 15:35:36][INFO][HMCTxnUtils][HMCTxnUtils.php:82][execBackgroundProcess]: Trying to background a new process, cluster=dev, txnId=41, command=/usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php, args= -c dev -d /var/db/hmc/data/data.db -a deploy -x 41, logFile=/var/log/hmc/hmc.txn.41.log, execCommand=/usr/bin/php /usr/share/hmc//php/util/BackgroundExecutor.php -t “41″ -c “/usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php” -a ” -c dev -d /var/db/hmc/data/data.db -a deploy -x 41″ -l “/var/log/hmc/hmc.txn.41.log”[2012:06:21 15:35:36][INFO][HMCTxnUtils][HMCTxnUtils.php:106][execBackgroundProcess]: Output from process, command=/usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php, txnId=41, output=Executing /usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php -c dev -d /var/db/hmc/data/data.db -a deploy -x 41 > /var/log/hmc/hmc.txn.41.log 2>&1

    [2012:06:21 15:35:36][INFO][HMCTxnUtils][HMCTxnUtils.php:106][execBackgroundProcess]: Output from process, command=/usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php, txnId=41, output=Executing /usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php -c dev -d /var/db/hmc/data/data.db -a deploy -x 41 > /var/log/hmc/hmc.txn.41.log 2>&1
    Background Child Process PID:6838
    Done with parent

    [2012:06:21 15:35:36][INFO][HMCTxnUtils][HMCTxnUtils.php:116][execBackgroundProcess]: Found child pid, command=/usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php, txnId=41, output=Executing /usr/bin/php /usr/share/hmc//php/orchestrator/ClusterMain.php -c dev -d /var/db/hmc/data/data.db -a deploy -x 41 > /var/log/hmc/hmc.txn.41.log 2>&1
    Background Child Process PID:6838
    Done with parent
    , pid=6838
    [2012:06:21 15:35:36][INFO][ClusterState][clusterState.php:19][updateClusterState]: Update Cluster State with {“state”:”DEPLOYMENT_IN_PROGRESS”,”displayName”:”Deployment in progress”,”timeStamp”:1340292936,”context”:{“txnId”:41}}
    [2012:06:21 15:35:36][INFO][ClusterMain:TxnId=41][ClusterMain.php:301][]: Taking action=deploy on cluster=dev, txn=41-0-0
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=HDFS, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=MAPREDUCE, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=DASHBOARD, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=GANGLIA, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=NAGIOS, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=MISCELLANEOUS, state=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:70][_deployHDP]: Deploying HDP with 6 services…. DryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:605][_installAllServices]: Installing HDP with 6 services… DryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-2-0:PENDING: Cluster install:PENDING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:1039][setState]: dev – INSTALLING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service HDFS …
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:HDFS_CLIENT][ServiceComponent.php:118][setState]: HDFS_CLIENT – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:HDFS_CLIENT][ServiceComponent.php:118][setState]: HDFS_CLIENT – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:156][install]: ServiceComponent NAMENODE is already INSTALLED!
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:156][install]: ServiceComponent NAMENODE is already INSTALLED!
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service HDFS complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service MAPREDUCE …
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:202][getServiceDependencies]: Found service with invalid state, service=HDFS, state=
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:MAPREDUCE_CLIENT][ServiceComponent.php:118][setState]: MAPREDUCE_CLIENT – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:MAPREDUCE_CLIENT][ServiceComponent.php:118][setState]: MAPREDUCE_CLIENT – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:156][install]: ServiceComponent JOBTRACKER is already INSTALLED!
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service MAPREDUCE complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service DASHBOARD …
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service DASHBOARD complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service GANGLIA …
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service GANGLIA complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service NAGIOS …
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service NAGIOS complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service MISCELLANEOUS …
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – INSTALLING dryRun=1
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – INSTALLED dryRun=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service MISCELLANEOUS complete.
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-2-0:PENDING: Cluster install:PENDING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:1039][setState]: dev – INSTALLED
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:690][_installAllServices]: Installing HDP with 6 services complete. DRYRUN=1
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:726][_startAllServices]: Starting HDP with 6 services…
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service HDFS …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-25-24:PENDING:HDFS start:PENDING
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-27-26:PENDING:NameNode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-27-26:PENDING:NameNode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-26-25:PENDING:Datanode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-26-25:PENDING:Datanode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:199][start]: ServiceComponent NAMENODE is already STARTED!
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:199][start]: ServiceComponent NAMENODE is already STARTED!
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-30-25:PENDING:Secondary NameNode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-30-25:PENDING:Secondary NameNode start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-25-24:PENDING:HDFS start:PENDING
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-32-24:PENDING:HDFS test:PENDING
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-32-24:PENDING:HDFS test:PENDING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service HDFS complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service MAPREDUCE …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-33-24:PENDING:MapReduce start:PENDING
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-35-33:PENDING:JobTracker start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-35-33:PENDING:JobTracker start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:199][start]: ServiceComponent JOBTRACKER is already STARTED!
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-37-33:PENDING:TaskTracker start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-37-33:PENDING:TaskTracker start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-33-24:PENDING:MapReduce start:PENDING
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-39-24:PENDING:MapReduce test:PENDING
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-39-24:PENDING:MapReduce test:PENDING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service MAPREDUCE complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service DASHBOARD …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-40-24:PENDING:Dashboard start:PENDING
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-41-40:PENDING:Monitoring Dashboard start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-41-40:PENDING:Monitoring Dashboard start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-40-24:PENDING:Dashboard start:PENDING
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – STARTED dryRun=1
    [2012:06:21 15:35:36][WARN][Service: DASHBOARD (dev)][Service.php:407][smoke]: Cannot find service-client node for DASHBOARD
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service DASHBOARD complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service GANGLIA …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-43-24:PENDING:Ganglia start:PENDING
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-44-43:PENDING:Ganglia Slave start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-44-43:PENDING:Ganglia Slave start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-45-43:PENDING:Ganglia Collector start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-45-43:PENDING:Ganglia Collector start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-43-24:PENDING:Ganglia start:PENDING
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – STARTED dryRun=1
    [2012:06:21 15:35:36][WARN][Service: GANGLIA (dev)][Service.php:407][smoke]: Cannot find service-client node for GANGLIA
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service GANGLIA complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service NAGIOS …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-47-24:PENDING:Nagios start:PENDING
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-48-47:PENDING:Nagios Server start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-48-47:PENDING:Nagios Server start:PENDING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – STARTED dryRun=1
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-47-24:PENDING:Nagios start:PENDING
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – STARTED dryRun=1
    [2012:06:21 15:35:36][WARN][Service: NAGIOS (dev)][Service.php:407][smoke]: Cannot find service-client node for NAGIOS
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service NAGIOS complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:807][startService]: Starting service MISCELLANEOUS …
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – STARTING dryRun=1
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – STARTED dryRun=1
    [2012:06:21 15:35:36][WARN][Service: MISCELLANEOUS (dev)][Service.php:407][smoke]: Cannot find service-client node for MISCELLANEOUS
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:810][startService]: Starting service MISCELLANEOUS complete. Result=0
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:735][_startAllServices]: Starting HDP with 6 services complete.
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=HDFS, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=MAPREDUCE, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=DASHBOARD, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=GANGLIA, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=NAGIOS, state=
    [2012:06:21 15:35:36][ERROR][OrchestratorDB][OrchestratorDB.php:127][getClusterServices]: Found service with invalid state, service=MISCELLANEOUS, state=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:70][_deployHDP]: Deploying HDP with 6 services…. DryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:605][_installAllServices]: Installing HDP with 6 services… DryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:610][persistTransaction]: persist: 41-2-0:INSTALLING: Cluster install:IN_PROGRESS
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:1039][setState]: dev – INSTALLING
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service HDFS …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: HDFS – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – NAMENODE – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – NAMENODE – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:118][setState]: NAMENODE – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – DATANODE – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – DATANODE – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:DATANODE][ServiceComponent.php:118][setState]: DATANODE – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – HDFS_CLIENT – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:HDFS_CLIENT][ServiceComponent.php:118][setState]: HDFS_CLIENT – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – HDFS_CLIENT – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:HDFS_CLIENT][ServiceComponent.php:118][setState]: HDFS_CLIENT – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:156][install]: ServiceComponent NAMENODE is already INSTALLED!
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAMENODE][ServiceComponent.php:156][install]: ServiceComponent NAMENODE is already INSTALLED!
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – SNAMENODE – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState HDFS – SNAMENODE – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:SNAMENODE][ServiceComponent.php:118][setState]: SNAMENODE – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: HDFS – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: HDFS (dev)][Service.php:130][setState]: HDFS – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service HDFS complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service MAPREDUCE …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MAPREDUCE – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – JOBTRACKER – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – JOBTRACKER – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:118][setState]: JOBTRACKER – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – MAPREDUCE_CLIENT – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:MAPREDUCE_CLIENT][ServiceComponent.php:118][setState]: MAPREDUCE_CLIENT – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – MAPREDUCE_CLIENT – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:MAPREDUCE_CLIENT][ServiceComponent.php:118][setState]: MAPREDUCE_CLIENT – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][ServiceComponent:JOBTRACKER][ServiceComponent.php:156][install]: ServiceComponent JOBTRACKER is already INSTALLED!
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – TASKTRACKER – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState MAPREDUCE – TASKTRACKER – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:TASKTRACKER][ServiceComponent.php:118][setState]: TASKTRACKER – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MAPREDUCE – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: MAPREDUCE (dev)][Service.php:130][setState]: MAPREDUCE – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service MAPREDUCE complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service DASHBOARD …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: DASHBOARD – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState DASHBOARD – DASHBOARD – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState DASHBOARD – DASHBOARD – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:DASHBOARD][ServiceComponent.php:118][setState]: DASHBOARD – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: DASHBOARD – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: DASHBOARD (dev)][Service.php:130][setState]: DASHBOARD – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service DASHBOARD complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service GANGLIA …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: GANGLIA – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState GANGLIA – GANGLIA_MONITOR – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState GANGLIA – GANGLIA_MONITOR – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR][ServiceComponent.php:118][setState]: GANGLIA_MONITOR – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState GANGLIA – GANGLIA_MONITOR_SERVER – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState GANGLIA – GANGLIA_MONITOR_SERVER – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:GANGLIA_MONITOR_SERVER][ServiceComponent.php:118][setState]: GANGLIA_MONITOR_SERVER – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: GANGLIA – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: GANGLIA (dev)][Service.php:130][setState]: GANGLIA – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service GANGLIA complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service NAGIOS …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: NAGIOS – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState NAGIOS – NAGIOS_SERVER – INSTALLING
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:577][setServiceComponentState]: Update ServiceComponentState NAGIOS – NAGIOS_SERVER – INSTALLED
    [2012:06:21 15:35:36][INFO][ServiceComponent:NAGIOS_SERVER][ServiceComponent.php:118][setState]: NAGIOS_SERVER – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: NAGIOS – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: NAGIOS (dev)][Service.php:130][setState]: NAGIOS – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service NAGIOS complete.
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:792][installService]: Installing service MISCELLANEOUS …
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MISCELLANEOUS – INSTALLING
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – INSTALLING dryRun=
    [2012:06:21 15:35:36][INFO][OrchestratorDB][OrchestratorDB.php:556][setServiceState]: MISCELLANEOUS – INSTALLED
    [2012:06:21 15:35:36][INFO][Service: MISCELLANEOUS (dev)][Service.php:130][setState]: MISCELLANEOUS – INSTALLED dryRun=
    [2012:06:21 15:35:36][INFO][Cluster:dev][Cluster.php:794][installService]: Installing service MISCELLANEOUS complete.
    [2012:06:21 15:35:36][INFO][PuppetInvoker][PuppetInvoker.php:247][genKickWait]: rm -f /etc/puppet/master/modules/catalog/files/modules.tgz
    [2012:06:21 15:35:36][INFO][PuppetInvoker][PuppetInvoker.php:250][genKickWait]: tar zcf /etc/puppet/master/manifestloader/modules.tgz /etc/puppet/master/modules
    [2012:06:21 15:35:37][INFO][PuppetInvoker][PuppetInvoker.php:253][genKickWait]: mv /etc/puppet/master/manifestloader/modules.tgz /etc/puppet/master/modules/catalog/files
    [2012:06:21 15:35:37][INFO][PuppetInvoker][PuppetInvoker.php:270][genKickWait]: Kick attempt (1/3)
    [2012:06:21 15:35:37][INFO][PuppetInvoker][PuppetInvoker.php:310][waitForResults]: Waiting for results from <hostname>
    [2012:06:21 15:35:37][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:35:42][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:35:47][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:35:52][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:35:57][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:02][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:07][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:12][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:17][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:22][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:27][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:32][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:37][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:42][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:47][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:52][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:36:57][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:02][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:07][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:12][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:17][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:22][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:27][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:32][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:37][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:42][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:42][WARN][PuppetInvoker][PuppetInvoker.php:322][waitForResults]: Kick timed out, waited 120 seconds
    [2012:06:21 15:37:43][INFO][PuppetInvoker][PuppetInvoker.php:270][genKickWait]: Kick attempt (2/3)
    [2012:06:21 15:37:43][INFO][PuppetInvoker][PuppetInvoker.php:76][sendKick]: <hostname>previous kick still running, will continue to wait
    [2012:06:21 15:37:43][INFO][PuppetInvoker][PuppetInvoker.php:310][waitForResults]: Waiting for results from <hostname>
    [2012:06:21 15:37:43][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:48][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:53][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:37:58][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:03][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:08][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:13][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:18][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:23][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:28][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:33][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:38][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:43][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:48][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:53][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:38:58][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:03][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:08][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:13][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:18][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:23][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:28][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:33][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:38][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:43][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:48][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:48][WARN][PuppetInvoker][PuppetInvoker.php:322][waitForResults]: Kick timed out, waited 120 seconds
    [2012:06:21 15:39:49][INFO][PuppetInvoker][PuppetInvoker.php:270][genKickWait]: Kick attempt (2/3)
    [2012:06:21 15:39:50][INFO][PuppetInvoker][PuppetInvoker.php:76][sendKick]: <hostname>previous kick still running, will continue to wait
    [2012:06:21 15:39:50][INFO][PuppetInvoker][PuppetInvoker.php:310][waitForResults]: Waiting for results from <hostname>
    [2012:06:21 15:39:50][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:39:55][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0
    [2012:06:21 15:40:00][INFO][PuppetInvoker][PuppetInvoker.php:314][waitForResults]: 0 out of 1 nodes have reported for txn 41-2-0

Viewing 11 replies - 1 through 11 (of 11 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #6295

    Annie Wong
    Member

    Started over with a fresh CentOS 5.8 install, and got the HDP install to go through by doing “yum install -y net-snmp net-snmp-utils” prior to doing “service hmc start”.

    Thx for the prompt help!

    Collapse
    #6227

    Sasha J
    Moderator

    Hi Annie,

    if you have a fresh CentOS 5.8, you can just try the steps from the beginning – without the additional steps mentioned in the thread

    You can find the steps here (the same page you are redirected to after initially downloading the HDP)

    http://hortonworks.com/download/thankyou_hdp1a/

    let us know if you run into any issues

    Sasha

    Collapse
    #6226

    Annie Wong
    Member

    Just tried with CentOS 5.8, the HDP install gets further now (eg I see it trying to yum stuff, which wasn’t happening earlier), however, the install fails. Found snmp errors like the ones reported in http://hortonworks.com/community/forums/topic/dependency-error-snmp/ in puppet_apply.log.

    I tried doing “yum install -y net-snmp net-snmp-utils” as suggested in that thread, but yum reports those pkgs are already installed:

    Package 1:net-snmp-5.3.2.2-17.el5_8.1.x86_64 already installed and latest version
    Package 1:net-snmp-utils-5.3.2.2-17.el5_8.1.x86_64 already installed and latest version
    Nothing to do

    Any ideas? Thx.

    Collapse
    #6223

    Annie Wong
    Member

    Thanks, will give it a shot.

    Collapse
    #6219

    Sasha J
    Moderator

    Annie,
    I did some research on your problem and found out that there is some discrepancies when running HDP on RHEL 5.7 and CentOS 5.7. I end up with the exact situation you are.
    It seems like you have to move to RHEL 5.8, which is fully supported and works fine.
    Please, rebuild your node with 5.8 and install HDP on it, it should work fine for you.
    Sorry for confusion…

    Collapse
    #6184

    Annie Wong
    Member

    Ended up updating PATH in /etc/init.d/functions to include /usr/local/bin/, and that fixed the ‘command not found’ error for ssh and scp.

    I also had to set PermitUserEnvironment in /etc/ssh/sshd_config and set PATH in ~root/.ssh/environment to get rid of the ‘command not found’ error for the ‘service’ cmd (since the ‘service’ cmd seemed to be invoked via ssh).

    My HDP install is still getting stuck though, with similar logs as the ones I included at the end of my first post. Is there any other log I can check? I looked at hmc.txn.*.log and that just had:

    tar: Removing leading `/’ from member names

    Also, found an SSL error in /var/log/puppet_agent_http.log,but subsequent ‘accepts’ didn’t error:

    [2012-06-21 17:04:45] DEBUG accept: 10.64.128.180:45092
    [2012-06-21 17:04:45] ERROR OpenSSL::SSL::SSLError:
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `call’
    /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread’
    /usr/lib/ruby/1.8/webrick/server.rb:95:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `each’
    /usr/lib/ruby/1.8/webrick/server.rb:92:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:23:in `start’
    /usr/lib/ruby/1.8/webrick/server.rb:82:in `start’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen’
    /usr/lib/ruby/1.8/thread.rb:135:in `synchronize’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:127:in `listen’
    /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:142:in `start’
    /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start’
    /usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:354:in `main’
    /usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:311:in `run_command’
    /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run’
    /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:410:in `hook’
    /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run’
    /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:401:in `exit_on_fail’
    /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run’
    /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute’
    /usr/bin/puppet:4
    [2012-06-21 17:04:45] DEBUG close: 10.64.128.180:45092
    [2012-06-21 17:05:28] DEBUG accept: 10.64.128.180:45096
    [2012-06-21 17:05:28] DEBUG Puppet::Network::HTTP::WEBrickREST is invoked.
    [2012-06-21 17:05:29] <hostname> – - [21/Jun/2012:17:05:28 PDT] “PUT /production/run/<hostname> HTTP/1.1″ 200 478
    [2012-06-21 17:05:29] – -> /production/run/<hostname>
    [2012-06-21 17:05:29] DEBUG close: 10.64.128.180:45096

    Thx.

    Collapse
    #6170

    Annie Wong
    Member

    Yup, had run those cmds as root.

    Collapse
    #6168

    All the initial setup steps “yum install…, service hmc start” need to be run as root. After that the HMC internally will manage the orchestration.

    Collapse
    #6166

    Annie Wong
    Member

    Thx for the quick reply.

    I’m currently trying to install HDP on just one node (the same node that HMC is running on).

    I tried setting $PATH in /etc/profile, but that didn’t make a difference. I’ll check with my company’s Ops team on this and the RHEL install.

    To help with debugging, could you clarify if the HMC scripts are being run as ‘puppet’ user?

    Thx.

    Collapse
    #6165

    Sasha J
    Moderator

    Just to clarify couple more things…
    On clean RHEL installation, ssh located in /usr/bin, not in /usr/local/bin, as well as scp.
    Please, check your system, it seems like it was build in some incorrect way…

    Collapse
    #6152

    Sasha J
    Moderator

    I believe you should rollback your installation and make sure that your global $PATH variable set correctly.
    Please, do so and then follow documentation precisely. You do not need to “hack” any of the scripts, it all works on correctly installed system.
    Also, how many nodes you are installing? Make sure that all names are resolvable correctly.

    Collapse
Viewing 11 replies - 1 through 11 (of 11 total)