Home Forums HDP on Linux – Installation Ganglia Monitor component remains down after installation of HDP2

This topic contains 5 replies, has 3 voices, and was last updated by  Victor Chugunov 5 months, 1 week ago.

  • Creator
    Topic
  • #46178

    Vinay Sudhakaran
    Participant

    Hi,

    I’ve been able to successfully install all the components of HDP2, except the Ganglia Monitor process on a single node Linux VM with CentOS 6.5
    Summary:
    Hostname:
    localhost.localdomain
    IP Address:
    <My IP Address>
    OS:
    centos6 (x86_64)
    CPU:
    2
    Disk:
    Data Unavailable
    Memory:
    5.71GB
    Load Avg:

    Agent
    Heartbeat:
    less than a minute ago

    Ganglia Server: Started
    Ganglia Monitors: 0/1 Ganglia Monitors Live

    The Alerts and Health checks display:
    Ganglia Monitor process for [Slaves, Resource Manager, NameNode, HistoryServer, HBase Master]
    Connection refused

    Any idea why the Ganglia Monitor process is not starting? Any recommendations to get this working would be helpful.
    I edited the gmond.conf and gmetad.conf to add my cluster name. Also, changed the hostname to localhost.localdomain but in vain.

    Regards,
    VS

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #50697

    Victor Chugunov
    Participant

    I have similar configuration and similar problems:
    Ganglia Server: Started
    Ganglia Monitors: 0/1 Ganglia Monitors Live

    The Alerts and Health checks display:
    Ganglia Monitor process for [Slaves, Resource Manager, NameNode, HistoryServer, HBase Master]
    Connection refused

    The suggested solution ( to modify the /etc/ganglia/hdp/HDP[*]/conf.d/gmond.master.conf files) doesn’t work for me – and they will be overwritten each time Ganglia Monitor is restarted. On the main dashboard page 4 metrics widgets (CPU Usage,Cluster Load, Memory Usage, Network Usage) have no data, only message “No Data. There was no data available. Possible reasons include inaccessible Ganglia service”. Ambari-server.log contains these statements:
    20:41:34,233 ERROR [pool-1-thread-14] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused
    20:41:34,239 ERROR [pool-1-thread-8] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused
    20:41:40,442 ERROR [pool-1-thread-23] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused
    20:41:40,447 ERROR [pool-1-thread-20] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused
    20:41:46,652 ERROR [pool-1-thread-15] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused
    20:41:46,657 ERROR [pool-1-thread-27] JMXPropertyProvider:487 – Caught exception getting JMX metrics : Connection refused

    Any idea why the Ganglia Monitor process is not starting? Any recommendations to get this working would be helpful.
    Thanks
    Victor

    Collapse
    #46195

    Jeff Sposetti
    Moderator

    Thanks Vinay. I think you only need to comment out the “bind = ” and that will work too.

    Collapse
    #46190

    Vinay Sudhakaran
    Participant

    Hi,

    I played around a bit with the hdp-gmond and hdp-gmetad conf files and was able to resolve the issue by commenting out the following lines in the /etc/ganglia/hdp/HDP[*]/conf.d/gmond.master.conf files

    /* The gmond cluster master must additionally provide an XML
    * description of the cluster to the gmetad that will query it.
    */
    /*
    tcp_accept_channel {
    bind = localhost.localdomain
    port = 8664
    }
    */

    Now, I have all the services up and ganglia reporting the metrics for them on a single node VM.

    Collapse
    #46180

    Vinay Sudhakaran
    Participant

    [root@localhost ~]# hostname -f
    localhost.localdomain
    [root@localhost ~]# cat /etc/hosts
    127.0.0.1 localhost.localdomain localhost
    ::1 localhost6.localdomain6 localhost6

    While installing HDP2, I had changed the ganglia user from ‘nobody’ to ‘vinay’.
    I updated that information in gmond.conf and gmetad.conf and since then Ganglia has been working. I can see the system metrics @http://localhost.localdomain/ganglia but in the HDP dashboard, the ganglia server and the monitor components don’t start. Hence, I continue to get this:

    The Alerts and Health checks display:
    Ganglia Monitor process for [Slaves, Resource Manager, NameNode, HistoryServer, HBase Master]
    Connection refused

    Thanks,
    VS

    Collapse
    #46179

    Jeff Sposetti
    Moderator

    Hi,

    1) What is in your /etc/hosts file?
    2) What does “hostname -f” return?

    Thanks!

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)