Home Forums HDP on Linux – Installation Ganglia not working on HDP 1.2

This topic contains 9 replies, has 4 voices, and was last updated by  Ardavan Moinzadeh 1 year, 4 months ago.

  • Creator
    Topic
  • #20007

    Pal J
    Participant

    I have successfully installed HDP on VM with CentOS 6.4 (64 bit) as OS, after successfully configuring the cluster via Ambari portal all services were started (showing green dot). (Note: both host and cluster are on same VM)
    On Dashboard tab/page “Cluster Metrics” were blank with message “No Data There was no data available. Possible reasons include inaccessible Ganglia Service“. Clicked on services tab to check “Ganglia” service, service was started with below messages

    Ganglia Collector [gmond] process down alert for HBase Master
    Ganglia Collector [gmond] process down alert for slaves
    Ganglia Collector [gmond] process down alert for NameNode
    Ganglia Collector [gmond] process down alert for JobTracker

    Checked Gangila service using following commands:
    “service gmetad status” the result was “gmetad (pid 6133) is running…”
    “service hdp- gmetad status” the result was =======================================
    Checking status of hdp-gmetad…
    =======================================
    “service gmond status “ the result was “gmond stopped”
    “service hdp- gmond status” the result was “Failed to find running /usr/sbin/gmond for cluster HDPSlaves”

    Clicked on “Host” tab and “Ganglia Monitor / Ganglia” was not started (red dot) tired to start by clicking on Action  Start but I was not successful

    The issue seems to Ganglia Monitor (gmond) not able to start on cluster HDPSlaves.
    Can you please advise what could be the cause for this issue and how to trouble to shoot this ?

Viewing 9 replies - 1 through 9 (of 9 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #28712

    Ardavan Moinzadeh
    Participant

    Sef, could you explain what does this error mean? I get the same error that Pal faced :

    ======================
    service hdp- gmond status” the result was “Failed to find running /usr/sbin/gmond for cluster HDPSlaves”
    =====================
    Thank you!

    Collapse
    #23896

    Seth Lyubich
    Keymaster

    Hi Pal,

    Thanks for letting us know that your issue is resolved.

    Seth

    Collapse
    #23879

    Pal J
    Participant

    Hi Seth
    Sorry for late respond. It took a while for me to new clean install. I installed HDP 1.2 on CentOS 6.4 all services including Ganglia are working

    thanks

    Collapse
    #20570

    Seth Lyubich
    Keymaster

    Hi Pal,

    There are several things that were suggested in the post in my last comment. Here is a summary that you can try:

    Make sure that time is synchronized in your cluster.

    directory /var/lib/ganglia/rrds should contain directories with rrd files.

    If you don’t have data there you might have issue with rrd tool. You can try the following:

    Make sure rrdcached process is running. Usually rrd tool gets started with hdp-gmetad service:

    #service hdp-gmetad start

    Starting hdp-gmetad…
    =============================
    /usr/bin/rrdcached already running with PID 24053
    /usr/sbin/gmetad already running with PID 24083

    If you have hdp-gmetad and hdp-gmond running make sure that corresponding ports are listening:

    netstat -anp| grep ’8660\|8661\|8662\|8663′

    make sure that sockets using IPv4 in output from command above.

    Finally, check that sockets are listening on expected configured ports (not localhost):

    grep -A4 8660 /etc/ganglia/hdp/gmetad.conf

    you should see something like below. Make sure that socket does not point to localhost.

    [root@ambari1 hdp]# grep -A4 8660 /etc/ganglia/hdp/gmetad.conf
    data_source “HDPSlaves” ambari1:8660
    data_source “HDPNameNode” ambari1:8661
    data_source “HDPJobTracker” ambari1:8662
    data_source “HDPHBaseMaster” ambari1:8663

    One more thing you can check is rrd tool packages. This is what I have on my machine:

    [root@ambari1 hdp]# rpm -qa|grep rrd
    rrdtool-1.4.5-1.el6.x86_64
    perl-rrdtool-1.4.5-1.el6.x86_64
    python-rrdtool-1.4.5-1.el6.x86_64

    Hope this helps,

    Thanks,
    Seth

    Collapse
    #20468

    Pal J
    Participant

    Hi Seth,
    Thanks for link..I also tired disabling IPV6 still no luck Gaglia service in Services tabs show green ..when I check the status of /etc/init.d/hdp-gmond start I get this message “Failed to start /usr/sbin/gmond for cluster HDPSlaves”
    I am new to HDP , please let me know what logs and do I have turn ON debug flags

    thanks
    Pal

    Collapse
    #20288

    Seth Lyubich
    Keymaster

    Hi Paj,

    Some debugging information in post below might be useful:

    http://hortonworks.com/community/forums/topic/lack-of-cluster-metrics-following-fresh-hdp-install-on-4-nodes/

    Hope this helps,
    Seth

    Collapse
    #20256

    Pal J
    Participant

    Hi Ted,
    Forgot to add below in the previous update
    On the “Hosts” page “Ganglia Monitor / Ganglia” was stopped I was not successful in starting it

    thanks

    Collapse
    #20255

    Pal J
    Participant

    Hi Ted,
    I killed “gmond” and “gmetad “ services using stop command and tired to start “Ganglia” in ambari initially I was not successful . I changed data_source from default “my cluster” to “localhost.localdomain” in /etc/ganglia/gmetad.conf and was able to start “Ganglia” in ambari. But there were still 4 errors in “Alerts and Health Check” one of them was “Ganglia Collector [gmond] process down alert for slaves”
    When I checked status of “service gmond” it was stopped and status of “service hdp-gmond “ was “Failed to find running /usr/sbin/gmond for cluster HDPSlaves”. The status of “gmetad” was OK

    Can you please advise what next…

    Collapse
    #20139

    tedr
    Member

    Hi Pal,

    Thanks for trying out Hortonworks Data Platform.

    What can sometimes happen that blocks the Ganglia services for HDP to come up is that when installed Ganglia puts its own hooks into system startup. The fix for this is to kill all of the running gmond processes and gmetad processes. Then start Ganglia from within Ambari.

    Thanks,
    Ted.

    Collapse
Viewing 9 replies - 1 through 9 (of 9 total)