Home Forums HDP on Linux – Installation Ambari agent does not work in installation

This topic contains 7 replies, has 2 voices, and was last updated by  Begum Tuncer 5 months ago.

  • Creator
    Topic
  • #48789

    Begum Tuncer
    Participant

    Hello,
    I tried to perform an automated installation on a cluster but even after the fifth trial it failed again with the same stange error. The hosts are registered successfully, but when the wizard installs the services it ends up with some warnings (yellow) in most of the hosts about hdfs&mapreduce checks and starting services. It shows the namenode host green, but in fact no services are installed in this host.
    Ambari server or agent logs tells me nothing remarkable:

    From ambari-server.log:
    11:55:05,546 INFO [Thread-30] ActionScheduler:434 – Not a service component, assuming its an action
    org.apache.ambari.server.ServiceComponentNotFoundException: ServiceComponent not found, clusterName=Example, serviceName=HDFS, serviceComponentName=HDFS_SERVICE_CHECK
    at org.apache.ambari.server.state.ServiceImpl.getServiceComponent(ServiceImpl.java:279)
    at org.apache.ambari.server.actionmanager.ActionScheduler.scheduleHostRole(ActionScheduler.java:429)
    at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:215)
    at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:137)
    at java.lang.Thread.run(Thread.java:662)
    12:05:18,611 WARN [Thread-30] ActionScheduler:187 – Operation completely failed, aborting request id:2

    From namenode’s ambari-agent.log:
    INFO 2014-02-17 11:35:56,638 security.py:51 – SSL connection established. Two-way SSL authentication is turned off on the server.
    INFO 2014-02-17 11:35:56,752 Controller.py:102 – Registered with the server with {u’exitstatus’: 0, u’response’: u’OK’, u’responseId’: 0, u’statusCommands’: []}
    INFO 2014-02-17 11:35:56,752 Controller.py:107 – Got status commands on registration []
    INFO 2014-02-17 11:35:56,752 Controller.py:258 – Response from server = OK
    INFO 2014-02-17 11:36:06,763 Heartbeat.py:76 – Sending heartbeat with response id: 0 and timestamp: 1392629766762. Command(s) in progress: False. Components mapped: False
    INFO 2014-02-17 11:36:09,036 HostCheckReportFileHandler.py:43 – Host check report at /var/lib/ambari-agent/data/hostcheck.result
    INFO 2014-02-17 11:36:09,036 HostCheckReportFileHandler.py:104 – Removing old host check file at /var/lib/ambari-agent/data/hostcheck.result
    INFO 2014-02-17 11:36:09,037 HostCheckReportFileHandler.py:109 – Creating host check file at /var/lib/ambari-agent/data/hostcheck.result
    INFO 2014-02-17 11:36:09,057 Controller.py:198 – No commands sent from the Server.

    So when the installation of services ends up with warnings, ambari ui stucks as “Loading” if i try to continue and complete installation. At that time, I get some logs like this in ambari-server.log:
    14:57:25,059 ERROR [pool-2-thread-8] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused

    How can I complete the installation of all services successfully in all hosts? Why is ambari-agent of some hosts work correctly but others do nothing after registering the host?

    OS: Centos 6.5
    Ambari version: 1.4.4.23-1
    Stack: HDP 1.3.2

Viewing 7 replies - 1 through 7 (of 7 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #49341

    Begum Tuncer
    Participant

    After some more try, i realized that the problem is general. I made the installation in completely different environment (fully fresh installation of Centos 6.5 virtual machines) and the problem occured very similarly. This time, none of the services are installed. Here is the complete ambari-server.log:

    http://dpaste.com/hold/1665379/

    In the same environment I made a successful installation when I choose HDP 1.3.3 stack version.

    I think there is problem between HDP 1.3.2 version and ambari-server/agent 1.4.4 version.

    Collapse
    #49150

    Begum Tuncer
    Participant

    Hi again,
    I think I can not explain the situation good.
    jdk-6u31-linux-x64.bin is correctly downloaded to ambari-server machine when i do “ambari-server setup”. I can see it under /var/lib/ambari-server/resources/.
    I start HDP installation, it finished with some warnings.
    I lookeed at all the hosts (There are 11 slaves and 4 masters). All hosts installed necessary services except one master (namenode) and one slave.
    jdk is installed on all machines, except these two machines.
    root user runs ambari-agent services on all machines.
    There are no error logs in these two machines ambari-agent.logs.
    I’ll try to install jdk’s manually on these two machines and try the HDP installation again.

    Collapse
    #49110

    Kenny Zhang
    Moderator

    Hi Begum,

    Yes, I think we need to fix the jdk problem first. The jdk is downloaded during you run “ambari-server setup” on the ambari server. You can verify this by looking at /var/lib/ambari-server/resources/jdk-6u31-linux-x64.bin.

    If you fails to download the jdk.bin, ambari will not be able to install it on all the hosts, which result in the HDP installation fail. Please also ensure you are running ambari-agent as root and has the privilege to install application to the system.

    Hope this helps.

    Thanks,
    Kenny

    Collapse
    #49078

    Begum Tuncer
    Participant

    Hi,
    I do ambari-server reset & setup but it did not help.
    I realized that on the problematic hosts, there is no HDP repo in /etc/yum.repos.d
    So I tried manually installing ambari server and agents, and manually copying repos to all hosts and then start installation. But again the same problem, no services install on the problematic hosts. In fact, “nothing” happened on them during the installation, i got the same logs as i sent before.
    Any ideas?

    Collapse
    #48962

    Kenny Zhang
    Moderator

    Hi Begum,

    This is interesting and make sense.
    Since the jdk is not even installed, so the error (JMX connection refused) is expected.
    Could you please issue the following commands and let me know what’s the result?
    # ambari-server reset
    # ambari-server setup

    You can refer to the following document:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.2/bk_using_Ambari_book/content/ambari-chap2-2.html

    Please let me know if you can successfully install JDK.

    Thanks,
    Kenny

    Collapse
    #48925

    Begum Tuncer
    Participant

    Hi Kenny,
    Thanks for reply.
    It seems that none of the services are installed on namenode host. Even jdk is not installed.
    # /usr/jdk64/jdk1.6.0_31/bin/jps
    -bash: /usr/jdk64/jdk1.6.0_31/bin/jps: No such file or directory
    # rpm -qa | grep hadoop
    #
    Only ambari-agent is installed on that host, nothing else.

    Collapse
    #48843

    Kenny Zhang
    Moderator

    Hi Begum,

    Could you please share the output of the following commands from the namenode host?

    # cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid
    # /usr/jdk64/jdk1.6.0_31/bin/jps

    Thanks,
    Kenny

    Collapse
Viewing 7 replies - 1 through 7 (of 7 total)