Home Forums HDP on Linux – Installation Can't handle ServiceComponentHostEvent event at current state

This topic contains 6 replies, has 2 voices, and was last updated by  Matt Harrington 9 months, 2 weeks ago.

  • Creator
    Topic
  • #28883

    Installation is consistently failing with yellow dashes. The first error in ambari-server.log is

    15:00:49,926 ERROR ServiceComponentHostImpl:721 - Can't handle ServiceComponentHostEvent event at current state, serviceComponentName=SQOOP, hostName=msh-hdpslave101, currentState=INSTALLED, eventType=HOST_SVCCOMP
    15:00:49,927 WARN HeartBeatHandler:233 - State machine exception
    org.apache.ambari.server.state.fsm.InvalidStateTransitionException: Invalid event: HOST_SVCCOMP_OP_IN_PROGRESS at INSTALLED
    at org.apache.ambari.server.state.fsm.StateMachineFactory.doTransition(StateMachineFactory.java:297)
    at org.apache.ambari.server.state.fsm.StateMachineFactory.access$300(StateMachineFactory.java:39)
    at org.apache.ambari.server.state.fsm.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:440)

    There are no errors in ambari-agent.log on the afore mentioned host msh-hdpslave101.

    Registration is successful and several components successfully install before this error appears. After this point, all components on all hosts fail to install with yellow dashes. Attempting a retry does not allow installation to proceed any farther.

    I am following a combination of the documentation http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.0/bk_using_Ambari_book/content/ambari-chap1-1.html and http://hortonworks.com/kb/ambari-on-ec2/ . All firewalling is disabled. This particular disk image (based on rhel6) has all of the prereqs configured and was previously used to successfully spinup a cluster. This problem has been encountered twice, each with fresh hosts.

    I can post any additional logs as well as teardown/rebuild the cluster to assist in diagnosis. Thanks!


    Red Hat Enterprise Linux Server release 6.4 (Santiago)
    ambari-server-1.2.3.7-1.noarch
    ambari-1.x-1.el6.noarch

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #28919

    Tedr, I gave up on figuring out the ports a while back and opened up all traffic between systems in the aws security policy.

    I have tried adding a few nodes and they all failed at the same point. It appears to be sqoop again, just like in the very first error message.

    [check] DataNode install
    [check] Ganglia Monitor install
    [check] HBase Client install
    [check] HBase RegionServer install
    [check] HCat install
    [check] HDFS Client install
    [check] Hive Client install
    [check] MapReduce Client install
    [check] Oozie Client install
    [check] Pig install
    [dash] Sqoop install
    [dash] TaskTracker install
    [dash] ZooKeeper Client install

    Retrying resulted in a success, but sqoop is no longer in the list of items to install:

    [check] DataNode start
    [check] Ganglia Monitor start
    [check] HBase RegionServer start
    [check] TaskTracker start

    I don't think I need sqoop at this time, but it looks like it never gets installed.

    Collapse
    #28909

    tedr
    Moderator

    Hi Matt,

    Haven’t had the chance to look over the log you posted yet, but the only bit that might have some minor issues is that when installing on EC2 you need to make sure that the AWS security policy allows the boxes to connect to each other in addition to turning off the local firewalls.

    Thanks,
    Ted.

    Collapse
    #28903

    After lunch I hit retry again and the installation completed. There were no configuration changes between now and then, so I guess this points to either an intermittent system/network issue or perhaps ambari installing things in the wrong order (and needing a certain number of retry clicks to get through it all.) I would really like to know your process for debugging this sort of problem to help me fix it going forward.

    Collapse
    #28902

    Given the scope of the installation failure, let me post a few sanity checks on the network.

    The network:

    [root@msh-ambarimaster101 ~]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6

    ..external ips removed..

    10.99.0.206 msh-ambarimaster101
    10.99.0.207 msh-nagios101
    10.99.1.168 msh-hdpmaster101
    10.99.1.167 msh-hdpmaster102
    10.99.1.169 msh-hdpslave101
    10.99.1.171 msh-hdpslave102
    10.99.1.170 msh-hdpslave103

    Keyed ssh and consistent /etc/hosts files:

    [root@msh-ambarimaster101 ~]# md5sum /etc/hosts
    63eaa9d6000ce0ca1916cee345cd5406 /etc/hosts
    [root@msh-ambarimaster101 ~]# ssh msh-hdpmaster101 md5sum /etc/hosts
    63eaa9d6000ce0ca1916cee345cd5406 /etc/hosts

    Local firewalls disabled:

    [root@msh-ambarimaster101 ~]# chkconfig --list iptables
    iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
    [root@msh-ambarimaster101 ~]# service iptables status
    iptables: Firewall is not running.

    [root@msh-ambarimaster101 ~]# ssh msh-hdpmaster101 chkconfig --list iptables
    iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
    [root@msh-ambarimaster101 ~]# ssh msh-hdpmaster101 service iptables status
    iptables: Firewall is not running.

    Collapse
    #28901

    Thanks for your quick response. I have uploaded msh-ServiceComponentHostEvent-ambari-server.log. It is hard to tell specifically what service fails at this point, because I hit retry and that removed all entries that had succeeded.

    On the master, the first master node, there are orange dashes by pig, sqoop, tas
    ktracker, and zookeeper client and server.

    On the second master, the dashes are with mysql server, oozie client and server,
    pig, snamenode, sqoop, tasktracker, webhcat, and zookeeper client and server.

    One slave node failed on everything it looks like, and the other two are just mi
    ssing sqoop, tasktracker, and the zookeepers.

    My nagios node is separate and that completely failed as well.

    Collapse
    #28896

    tedr
    Moderator

    Hi Matt,

    thanks for using trying out HDP/Ambari. Which component is failing to install? Also can you post the complete ambari-server.log to our FTP site http://ftp.hortonworks.com user/pass=dropoff/horton. try ti name it something unique to you so that we can tell it’s your log. You won’t be able to see any files there just upload.

    Thanks,
    Ted.

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)