Home Forums HDP on Linux – Installation Ambari – Cluster restart

This topic contains 13 replies, has 5 voices, and was last updated by  tedr 1 year, 1 month ago.

  • Creator
    Topic
  • #15520

    Hi,

    It is a new install consisting of two node cluster. Install was done through Ambari. Everything went fine.

    It was a unplanned reboot of nodes while cluster was up and running. After system came up still Ambari thinks all components are running. So I tried to stop Namenode, Mapreduce, Hive etc.. But Ambari is not shutting down services and I see red icon is blinking.

    How do I tell Ambari that everything (all services) is down and just restart it?

    Thanks.

Viewing 13 replies - 1 through 13 (of 13 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #27464

    tedr
    Moderator

    Hi Lance,

    since we are working on this via email, let’s carry on our conversation there and then update this thread with the solution.

    Thanks,
    Ted.

    Collapse
    #27444

    Lance Fang
    Participant

    Hi Ted,

    Thank you so much on your diligence. After setting these environment variables and successfully executing the command below, the dashboard is still “flashing RED” and “start” button is still gray out in the services tab. Appreciate anymore insights…

    Collapse
    #27391

    tedr
    Moderator

    Hi Lance,

    Email sent. The reason it complained about the missing URL is that when you copied and pasted you grabbed only the first line of the command. The url it needs is the second line shown. which must be copied as well.

    Thanks,
    Ted.

    Collapse
    #27324

    Lance Fang
    Participant

    Ted,

    I also noticed problem when cut and paste from the software forum. Tried multiple combinations and still resulted with this error.

    [root@dwbi-durl-gphd0 ~]# curl –user admin:$PASSWORD -i -X PUT -d’{“ServiceInfo”:{“state”:”INSTALLED”}}’
    curl: no URL specified!
    curl: try ‘curl –help’ or ‘curl –manual’ for more information
    [root@dwbi-durl-gphd0 ~]#

    Maybe you can email the command to me at lance.fang@emc.com

    Collapse
    #27311

    tedr
    Moderator

    Hi Lance,

    Oops, I see why it’s asking you for the password. there is a small problem with our forum software, it will merge two dashes into what looks like one so the ‘user’ word in the command needs to be prefaced with two dashes not just one. If there is just one entered the command will pass the user ‘ser’ and not pass a password. but if you preface it with the required two it will pass the user ‘admin’ with the password from the env variable.

    Thanks,
    Ted.

    Collapse
    #27305

    Lance Fang
    Participant

    Ted,

    Here is what I have set and sourced on the ambari-server.

    export PASSWORD=admin
    export HOST=dwbi-durl-gphd0.gso.lab.emc.com
    export CLUSTER_NAME=POC
    export SERVICE_NAME=NAGIOS

    When trying to execute the curl command it prompted me for host password. I have tried all password without success.

    [root@dwbi-durl-gphd0 ~]# curl -user admin:$PASSWORD -i -X PUT -d ‘{“ServiceInfo”:{“state”:”INSTALLED”}}’
    Enter host password for user ‘ser’:
    curl: (6) Couldn’t resolve host ‘admin:admin’
    [root@dwbi-durl-gphd0 ~]#

    Collapse
    #27299

    tedr
    Moderator

    Hi Lance,

    No, SERVICE_NAME is one service at a time the command needs to be executed once for each service. Also these commands only need to be done on the box that ambari-server is on.

    Thanks,
    Ted.

    Collapse
    #27297

    Lance Fang
    Participant

    Ted,

    Thank you …I just restarted all ambari agents and then server. Still flashing red on the services ..

    Do I set these environment variables on all nodes?

    I assume the “SERVICE_NAME” is on one line with comma separated?

    Collapse
    #27274

    tedr
    Moderator

    Hi Lance,

    Have you restarted ambari-agent on all nodes? If you have and there are no background operations pending you can use the following commands from the shell to reset the statuses:

    set the following environment variables:
    PASSWORD=your ambari admin password
    HOST=the name of the server on which ambari-server is running
    CLUSTER_NAME=the name you gave to the cluster during install
    SERVICE_NAME= one of NAGIOS, GANGLIA, HIVE, HCATALOG, HBASE, OOZIE, ZOOKEEPER, WEBHCAT, MAPRED, HDFS (listed in the best order to shut them all off, and they are case sensitive, so exactly as typed. also you will need to change this variable for each service)

    to set the services to stopped state:
    curl –user admin:$PASSWORD -i -X PUT -d ‘{“ServiceInfo”: {“state” : “INSTALLED”}}’ http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$SERVICE_NAME

    After getting them to a stopped state you should be able to start them back up in the ambari ui.

    Thanks,
    Ted.

    Collapse
    #27265

    Lance Fang
    Participant

    I have a cluster of 5 nodes and due to unplanned network outage, the servers were also rebooted. Now all services are flashing red on the dashboard and won’t let me restart any of the services

    Here is the version I am using:

    Name : ambari-agent
    Arch : x86_64
    Version : 1.2.2.5

    Any points are appreciated to get me back to normality.

    Collapse
    #16207

    Seth Lyubich
    Keymaster

    HI Krishnasamy,

    Thanks for letting us know that the issues is fixed.

    Seth

    Collapse
    #16203

    Thanks for the reply. After restart I’d to start ambari-agent which was not running. All is fine now.

    Collapse
    #15555

    tedr
    Member

    Hi Krishnasamy,

    What version of Ambari are you running on? You can find out with "yum info ambari-agent". if you are running on version 1.2.0 I would suggest that you update to 1.2.1 following the instructions in chapter 7 of our online documents. In version 1.2.1 Ambari will catch up to show what is actually running in a minute or so.

    Thanks,
    Ted.

    Collapse
Viewing 13 replies - 1 through 13 (of 13 total)