Home Forums HDP on Linux – Installation Cold ShutDown And Re-Start Procedure

This topic contains 10 replies, has 3 voices, and was last updated by  Larry Liu 1 year, 9 months ago.

  • Creator
    Topic
  • #14654

    Stephen Bovy
    Member

    I am using HDP 1.2

    I have “successfully” created a (1) Instance VM On Windows VM-W 8.0

    Based on docs I have tried to create a shutdown script and a start-up script

    But the docs do not mention how to Integrate Ambari into the shutdown and start up procedure

    Getting these scripts to work robustly and reliably is a crap-shoot

    And after a re-boot the state and status of ambari after re-starting all components is bogus

    PLEASE provide more complete and robustly reliable cold-shutdown and cold start up procedures

    Usage of a VM without such procedures is a pain in the but, thanks

    The same issues are probably TRUE for using a cluster in-the-cloud :)

Viewing 10 replies - 1 through 10 (of 10 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #14716

    Larry Liu
    Moderator

    Here is my email: lliu@hortonworks.com

    Collapse
    #14692

    Stephen Bovy
    Member

    Thanks for engaging me in this discussion

    Yes of course the doc is not perfect

    But our purpose here is to improve the doc and move foreward :)

    I am posting my experience in the hopes that we can gain some insights and get the improvements

    Here is the error ( already previously posted )

    HIVE-METASTORE status check
    CRIT for less than a minute
    CRITICAL: Error accessing hive-metaserver status [WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

    Current Status:
    CRITICAL
    (for 0d 0h 49m 51s)
    Status Information: CRITICAL: Error accessing hive-metaserver status [WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    Hive history file=/tmp/nagios/hive_job_log_nagios_201301301847_1721941728.txt
    FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask]

    >>>>>>>>

    So all I can say thus far is that manual shut-down ( or catastophic hsutdown ?? )

    And automated start up does not play well with ambari and ganglia ( as it currently stands )

    If you like we can continue this discussion off-line and I can sent you the scripts that I have tried to use

    BTW I am with Terdata ( And I am trying to get this VM setup for future development purposes )

    But as I stated earlier VM machines are on/off demand machines that can not be guranteed to remain running
    for any length of time and we need robust shut down and re-start procedures that are not gui driven

    Thanks again for your assistence

    Collapse
    #14685

    Larry Liu
    Moderator

    The point for ambari is to manage the cluster in UI. The documentation is not perfect. Also your points to start services across the cluster is a good idea.

    What is the bug you find out in the HIVE log4j property matastore files?

    Larry

    Collapse
    #14684

    Stephen Bovy
    Member

    Thanks

    Use the GUI ?? ( Are You kidding ?? )

    We are using LInux here where scripted “automation” is the KING !!

    And this type of shutdown and restart MUST be done by an automated SCRIPT

    Doing these type of procedures from the GUI is rediculously laim and totally un-acceptable

    BTW >> I already stated in my previous posts that I am ALREADY following the manual start up and shutdown

    procedures in the reference book you just pointed me to

    Those procedures are inadequet for the following reasons

    1) They do not mention when and where the ambari-server and its agents should be shut down and re-started

    After following these procedures the interaction and interfaces between ambari and ganglia is ambigous undefined
    and totally un-reliable

    Also after following these procedures I discovered that the log4j property matastore files for HIVE has syntax errors

    BTW these problems are not limited to 1.2

    The same problems occured when we were in the middle of the HortonWorks ADMIN class :)

    Basically after a machine shut-down and reboot ( and after all components have been successfully re-started )
    The ambari server can no longer get correct state info from ganglia and does not correctly reflect the re-started
    state of the components

    Collapse
    #14678

    Larry Liu
    Moderator

    If you really prefer doing it manually, here is the documentation:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.0/bk_reference/content/reference_chap3.html

    Collapse
    #14677

    Larry Liu
    Moderator

    Please use ambari to stop/start services.

    You can try to stop the services from ambari web UI and start the service after reboot.

    Collapse
    #14676

    Stephen Bovy
    Member

    I did the installation using the “ambari” install method

    I did not use the manual install method

    The install works flawlessy ( I have done it 3-4 times) already :)

    The problem comes when I try to shutdown and then restart

    I have written a shutdown and restart script ( based on the “manual” ) shutdown and restart procedires mentioned
    in the reference manual

    After shutdown and restart the interraction between ambari and Ganglia seems to be “dead” as a door-nail

    Frankly guys ( to me this is “basic” ) The very first thing I TEST is shutdown and start up procedures

    Because as a developer ( using vm technology ) I have an un-avoidable requirement to shut down and re-start

    Collapse
    #14670

    Larry Liu
    Moderator

    Hi, Stephen

    How did you start install the HDP 1.2?

    If you have manually install hadoop before you install HDP, please remove it. Or start from a fresh operating system installation.

    Larry

    Collapse
    #14661

    Stephen Bovy
    Member

    Hi Sef, thanks for responding

    This is a ( 1 ) instance CentOS 6.3 VM

    The hadoop Refference has a section fro manual shutdown and Re-start

    I am using the information in that section to create a shut-down script and a re-start script

    I think part of the problem is trying to get Ganglia Started correctly

    Ohh BTW-FYI After shuting Down And re-starting the following error poped out >>

    HIVE-METASTORE status check
    CRIT for less than a minute
    CRITICAL: Error accessing hive-metaserver status [WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

    Current Status:
    CRITICAL
    (for 0d 0h 49m 51s)
    Status Information: CRITICAL: Error accessing hive-metaserver status [WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    Hive history file=/tmp/nagios/hive_job_log_nagios_201301301847_1721941728.txt
    FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask]

    Collapse
    #14659

    Seth Lyubich
    Keymaster

    Hi Stephen,

    Thanks for using HDP.

    Did you start ambari server after restart (ambari-server start)? If this does not resolve your issue can you please clarify:

    – “…successfullyā€¯ created a (1) Instance VM” – Is this a single node machine?

    – Based on which docs you created your scripts?

    – Ambari has UI functionality to start/stop services. Can you please clarify what you trying to do? Did you start Ambari server?

    – state and status of ambari after re-starting all components is bogus. Can you provide more details?

    Hope this helps,

    Thanks,
    Seth

    Collapse
Viewing 10 replies - 1 through 10 (of 10 total)