host registration failing with no errors logged

I’m testing an Ambari 1.2 installation on Centos 6. Currently I’m putting the Ambari server and agent on the same system to keep things simple at first.

The host registration progress fails according to the Web UI. But I don’t see any errors in the server or agent log.

The server log gets to this point, and then just logs the “Using resource provider” line repeatedly:

12:43:17,506 INFO BSHostStatusCollector:55 - Request directory /var/run/ambariserver/bootstrap/1
12:43:17,506 INFO BSHostStatusCollector:62 - HostList for polling on [ambari-server]
12:43:18,345 INFO ClusterControllerImpl:92 - Using resource provider org.apache.ambari.server.controller.internal.HostResourceProvider for request type Host


The agent is repeatedly logging this:

INFO 2013-03-02 12:47:54,280 Heartbeat.py:59 - Heartbeat dump: {'componentStatus': [],
'hostname': 'localhost.localdomain',
'nodeStatus': {'cause': 'NONE', 'status': 'HEALTHY'},
'reports': [],
'responseId': 27,
'timestamp': 1362257274279}
INFO 2013-03-02 12:47:54,332 Controller.py:150 - Got server response: {u'executionCommands': [],
u'registrationCommand': None,
u'responseId': 28,
u'restartAgent': False,
u'statusCommands': []}
INFO 2013-03-02 12:47:54,332 Controller.py:116 - No commands from the server : []
INFO 2013-03-02 12:47:54,332 Controller.py:116 - No commands from the server : []
INFO 2013-03-02 12:47:54,332 Controller.py:177 - No commands sent from the Server.


The only similar issue I’ve seen searching the forum warned that DNS must be set up, so I do have dnsmasq running to intercept local name lookups:

[root@ambari-server ~]# hostname
ambari-server
[root@ambari-server ~]# hostname -f
ambari-server
[root@ambari-server ~]# nslookup ambari-server
Server: 192.168.1.154
Address: 192.168.1.154#53

Name: ambari-server
Address: 127.0.0.1


Any help appreciated. I’ve uploaded the server and agent logs, and the output of ambari-check.sh, to the support FTP site as jbaxter-ambari-register-fail.tar.gz

Topic

[Sasha JModerator]

You can check for the key generated in /var/lib/ambari-agent/keys
key files have the same name and FQDN for host.

Sasha

[Sasha JModerator]

In a home it may depend on ISP…
I have Comcast at home and it push it’s domain explicitly, and even with “hostname -f” returns short name, domain still in there.
Check /etc/resolv.conf, it may show you the actual domain name it have.

Thank you!
Sasha

[Joel BaxterMember]

In the first situation that I described however (home test cluster), hostname -f returns the shortname, so would there be a longer FQDN for the host that could be seen/used by key generation, or any process? And I wasn’t seeing any validation errors in the logs… I’d think that a bad host key would result in some sort of visible error.

I’ll do some more digging in that general area. Check /etc/sysconfig/network, see if I have a domain search path set in resolv.conf, whatever else comes to mind.

[Sasha JModerator]

Joel,
this is not a coincident…
You have to use FQDN when adding hosts to the cluster.
It is always fails if you try to use short name.
The reason for it, is that ambari-agent generates host key file based on FQDN, so if you provide short name, key can not be accepted…

Thank you!
Sasha

[Joel BaxterMember]

So as I mentioned below, I’ll try a few more things on that host tonight, but for experiment’s sake I ran through the setup process again on a different host just now, a physical host that I overwrite with a new CentOS 6 installation.

This new host uses a real DNS server and does have NTP enabled.

The host is “bd-004.mycompany.com” and I’m trying to “add it to itself” to set up a one-node cluster on the same host as the Ambari server. If I add “bd-004″ then I get the same behavior as with the other host; failure on host registration, with no errors that I can see in the logs.

I cleaned up after that failure and tried again adding the host as “bd-004.mycompany.com” and this time it succeeded. I don’t know if that is a coincidence or not yet; I’ll do some more tests and look at the new logs later.

[Joel BaxterMember]

Thanks for the suggestions.

I’ll try the hosts file change tonight… I was pondering that too. And NTP too… why not, although it seems like an odd requirement for a single-host setup.

iptables is disabled.

[tedrMember]

Hi Joel,

Thanks for trying Hortonworks Data Platform.

In looking over the files that you uploaded I see a few possibilities:

* Try removing ‘ambari-server’ from the first two lines of your /etc/hosts file and then inserting the following line:
192.168.1.154 ambari-server
* Turn on NTP on all nodes on the cluster – the output of check.sh shows that ntpd is stopped and it is mentioned in the docs that this being on is a pre-requisite
* Have you turned off iptables?

Thanks,
Ted.

[Author]

Replies