Home Forums Hive / HCatalog WebHCat Server / WebHCat

Tagged: 

This topic contains 23 replies, has 7 voices, and was last updated by  Seth Lyubich 10 months ago.

  • Creator
    Topic
  • #21734

    Derek Wilson
    Member

    Hi,
    I have an installation of HDP 1.2. the WebHCat is stuck in Ambari on a flashing red light. How do I reset this? In Ambari I cannot start or stop the service.

    Thanks!

Viewing 23 replies - 1 through 23 (of 23 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #23956

    Seth Lyubich
    Keymaster

    Hi Ray,

    Thanks for letting us know that issue is resolved and your feedback regarding editor.

    Seth

    Collapse
    #23898

    Ray Roberts
    Participant

    Robert,

    Awesome, that worked.

    So, just an interesting side note, this editor must do some type of encoding to the text. Your command below was causing parsing errors. My colleague (who is a developer) found that both the single and double quotes were transformed somehow, so when I did a copy and paste, the command didn’t work.

    Thanks for your help, that flashing red was driving me crazy :)

    -Ray

    Collapse
    #23876

    Robert
    Participant

    Hi Ray,
    Shut down web hcat manually. Then try restarting the ambari-agent on the node where webhcat is running and verify using the api call the used in the previous post to check what the state is. (Check after about 5 min) If it is still “STOP_FAILED”, then try the following api call:

    within a terminal or console
    curl -i -u admin:admin -X PUT -d ‘{“ServiceInfo”: {“state”: “INSTALLED”} }’ http://localhost:8080/api/v1/clusters/cluster_name/services/WEBHCAT

    NOTE: admin:admin is default, if you changed it, please put the proper credentials. Also, cluster_name and localhost are specific to your cluster, hence make necessary changes to the above REST call.

    Then go ahead run the api call to get the actual state again (the tried in previous post). If the state is now INSTALLED, try to start the service via ambari. Make sure first the service is not running on the node and that ambari-agent has been restarted.

    Hope that helps.

    Regards,
    Robert

    Collapse
    #23861

    Ray Roberts
    Participant

    Robert,

    I can start/stop via the command below. I also verified the process is running.

    Here is each reference to webhcat in the output from json:

    “href” : “http://$HOSTNAME:8080/api/v1/clusters/$CLUSTERNAME/hosts/$HOSTNAMEl/host_components/WEBHCAT_SERVER”,
    “HostRoles” : {
    “cluster_name” : “$CLUSTERNAME”,
    “desired_state” : “INSTALLED”,
    “state” : “STOP_FAILED”,
    “component_name” : “WEBHCAT_SERVER”,
    “service_name” : “WEBHCAT”,
    “host_name” : “$HOSTNAME.DOMAIN.COM”,
    “configs” : { },
    “desired_configs” : { }
    },

    :\”1367270197\”,\”last_hard_state\”:\”2\”,\”last_time_ok\”:\”1365698197\”,\”last_time_warning\”:\”0\”,\”last_time_unknown\”:\”1365698257\”,\”last_time_critical\”:\”1367357737\”,\”is_flapping\”:\”0\”,\”last_check\”:\”1367357737\”,\”service_type\”:\”MAPREDUCE\”},{\”service_description\”:\”WebHCat Server status check\”,\”host_name\”:\”$HOSTNAME\”,\”current_attempt\”:\”3\”,\”current_state\”:\”2\”,\”plugin_output\”:\”(Return code of 127 is out of bounds – plugin may be missing)\”,\”last_hard_state_change\”:\”1367270101\”,\”last_hard_state\”:\”2\”,\”last_time_ok\”:\”1365698221\”,\”last_time_warning\”:\”0\”,\”last_time_unknown\”:\”0\”,\”last_time_critical\”:\”1367357761\”,\”is_flapping\”:\”0\”,\”last_check\”:\”1367357761\”,\”service_type\”:\”WEBHCAT\”},{\”service_description\”:\”ZooKeeper Server process down\”,\”host_name\”:\”$HOSTNAME\”,\”current_attempt\”:\”1\”,\”current_state\”:\”2\”,\”plugin_output\”:\”(Return code of 127 is out of bounds – plugin may be missing)\”,\”last_hard_state_change\”:\”1367270134\”,\”last_hard_state\”:\”2\”,\”last_time_ok\”:\”1365698254\”,\”last_time_warning\”:\”0\”,\”last_time_unknown\”:\”0\”,\”last_time_critical\”:\”1367357734\”,\”is_flapping\”:\”0\”,\”last_check\”:\”1367357734\”,\”service_type\”

    1367357722\”,\”service_type\”:\”NAGIOS\”},{\”service_description\”:\”Percent ZooKeeper Servers down\”,\”host_name\”:\”$HOSTNAME\”,\”current_attempt\”:\”1\”,\”current_state\”:\”1\”,\”plugin_output\”:\”(null)\”,\”last_hard_state_change\”:\”1367270155\”,\”last_hard_state\”:\”1\”,\”last_time_ok\”:\”1365698305\”,\”last_time_warning\”:\”1367357755\”,\”last_time_unknown\”:\”0\”,\”last_time_critical\”:\”1365267685\”,\”is_flapping\”:\”0\”,\”last_check\”:\”1367357755\”,\”service_type\”:\”ZOOKEEPER\”}],\”hostcounts\”:{\”up_hosts\”:0,\”down_hosts\”:14},\”servicestates\”:{\”PUPPET\”:0,\”HDFS\”:1,\”MAPREDUCE\”:1,\”HIVE-METASTORE\”:1,\”WEBHCAT\”:1,\”ZOOKEEPER\”:0}}”,

    Collapse
    #23856

    Robert
    Participant

    Hi Ray,
    Are you able to start the webhcat process outside of Ambari via the command:
    su – hcat -c ‘env HADOOP_HOME= /usr/lib/hcatalog/sbin/webhcat_server.sh start’

    View the pid that tis generated at /var/run/webhcat/webhcat.pid and run a ps -aux | grep so you can verify the process is up and running. If the process can start and stop without an issue. Then trying running the following from the url on your browser after logging into ambari

    http://:8080/api/v1/clusters//host_components?fields=HostRoles

    and look for the WEBHCAT_SERVER component within the json output and provide the output for WEBHCAT_SERVER here. You want to note the state and target state. Let us know what you find out.

    Regards,
    Robert

    Collapse
    #23833

    Ray Roberts
    Participant

    Larry,

    Yes, I do have that file.

    -Ray

    Collapse
    #23822

    Larry Liu
    Moderator

    Hi, Ray,

    One thing to check is if there is pid file in the following location.

    /var/run/webhcat/webhcat.pid

    Larry

    Collapse
    #23819

    Ray Roberts
    Participant

    Yi,

    Yes, zookeeper is installed in my cluster. I’ve had no issues with it.

    Here is my config from webhcat-site.xml:

    templeton.hcat
    /usr/bin/hcat

    templeton.override.enabled
    false

    templeton.libjars
    /usr/lib/zookeeper/zookeeper.jar

    templeton.hadoop.conf.dir
    /etc/hadoop/conf

    templeton.pig.path
    pig.tar.gz/pig/bin/pig

    templeton.hive.properties
    hive.metastore.local=false,hive.metastore.uris=thrift://$WEBHCATNODE:9083,hive.metastore.sasl.enabled=yes,hive.metastore.execute.setugi=true

    templeton.hive.archive
    hdfs:///apps/webhcat/hive.tar.gz

    templeton.hadoop
    /usr/bin/hadoop

    templeton.zookeeper.hosts
    $ZOOKEEPERNODE:2181

    templeton.jar
    /usr/lib/hcatalog/share/webhcat/svr/webhcat.jar

    templeton.exec.timeout
    60000

    templeton.pig.archive
    hdfs:///apps/webhcat/pig.tar.gz

    templeton.storage.class
    org.apache.hcatalog.templeton.tool.ZooKeeperStorage

    templeton.streaming.jar
    hdfs:///apps/webhcat/hadoop-streaming.jar

    templeton.hive.path
    hive.tar.gz/hive/bin/hive

    templeton.port
    50111

    Collapse
    #23789

    Yi Zhang
    Moderator

    Hi Ray,

    Is zookeeper installed in the cluster? can you check what is in webhcat-site.xml (or templeton-site.xml) what is configured for the zookeeper?

    Thanks,
    Yi

    Collapse
    #23786

    Ray Roberts
    Participant

    Seth,

    I did notice this error numerous times in the /var/log/webhcat/webhcat.log

    ERROR | 29 Apr 2013 16:16:03,106 | org.apache.hadoop.conf.Configuration | Failed to set setXIncludeAware(true) for parser org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@36b60b93:java.lang.Unsupport
    edOperationException: This parser does not support specification “null” version “null”
    java.lang.UnsupportedOperationException: This parser does not support specification “null” version “null”
    at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1123)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1109)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1045)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
    at org.apache.hcatalog.templeton.tool.JobStateTracker.getTrackingJobs(JobStateTracker.java:136)
    at org.apache.hcatalog.templeton.tool.ZooKeeperCleanup.getChildList(ZooKeeperCleanup.java:145)
    at org.apache.hcatalog.templeton.tool.ZooKeeperCleanup.run(ZooKeeperCleanup.java:102)

    Collapse
    #23755

    Ray Roberts
    Participant

    Seth,
    The ambari-log only shows the following in reference to webhcat:

    u’serviceName’: u’WEBHCAT’}
    INFO 2013-04-29 14:47:40,131 LiveStatus.py:124 – The live status for component WEBHCAT_SERVER of service WEBHCAT is None
    INFO 2013-04-29 14:47:40,132 ActionQueue.py:117 – Got live status for component WEBHCAT_SERVER of service WEBHCAT of cluster $CLUSTERNAME
    None
    INFO 2013-04-29 14:47:40,136 ActionQueue.py:93 – Took an element of Queue: {u’clusterName’: u’$CLUSTERNAME’,

    Also, I ran http://$CLUSTERNAME:8080/api/v1/clusters//requests/1?fields=tasks/* and received:

    “href” : “http://$CLUSTERNAME:8080/api/v1/clusters/$CLUSTERNAME/requests/1/tasks/41″,
    “Tasks” : {
    “exit_code” : 0,
    “stdout” : “notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully\nnotice: /Stage[2]/Hdp-hcat/Hdp::Package[hcat]/Hdp::Package::Process_pkg[hcat]/Package[hcatalog]/ensure: created\nnotice: /Stage[2]/Hdp-hcat/Hdp::Package[hcat]/Hdp::Package::Process_pkg[hcat]/Hdp::Java::Package[hcat]/File[/usr/jdk/jdk1.6.0_31/bin/java hcat]/ensure: created\nnotice: /Stage[2]/Hdp-hcat/Hdp::User[hcat]/User[hcat]/gid: gid changed ’488′ to ‘hadoop’\nnotice: /Stage[2]/Hdp-hcat/Hdp::Directory_recursive_create[/var/run/webhcat]/Hdp::Exec[mkdir -p /var/run/webhcat]/Exec[mkdir -p /var/run/webhcat]/returns: executed successfully\nnotice: /Stage[2]/Hdp-hcat/Hdp::Directory_recursive_create[/var/run/webhcat]/Hdp::Directory[/var/run/webhcat]/File[/var/run/webhcat]/owner: owner changed ‘root’ to ‘hcat’\nnotice: /Stage[2]/Hdp-hcat/Hdp::Directory_recursive_create[/var/run/webhcat]/Hdp::Directory[/var/run/webhcat]/File[/var/run/webhcat]/group: group changed ‘root’ to ‘hadoop’\nnotice: /Stage[2]/Hdp-hcat/Hdp-hcat::Configfile[hcat-env.sh]/Hdp::Configfile[/etc/hcatalog/conf/hcat-env.sh]/File[/etc/hcatalog/conf/hcat-env.sh]/ensure: created\nnotice: Finished catalog run in 28.11 seconds”,
    “status” : “COMPLETED”,
    “stderr” : “none”,
    “host_name” : “$CLUSTERNAMEl”,
    “id” : 41,
    “cluster_name” : “$CLUSTERNAME”,
    “attempt_cnt” : 1,
    “request_id” : 1,
    “command” : “INSTALL”,
    “role” : “HCAT”,
    “start_time” : 1365029100965,
    “stage_id” : 1

    Collapse
    #23504

    Seth Lyubich
    Keymaster

    Hi Ray,

    Can you please check ambari-agent.log on the server running WebHcat? Also, please make sure that ambari-agent is runing on WebHcat node.

    Also, can you please run this command:

    http://:8080/api/v1/clusters/

    and http://:8080/api/v1/clusters//requests/1?fields=tasks/*

    cluster_name can be found by running:

    http://:8080/api/v1/clusters/

    Thanks,
    Seth

    Collapse
    #23409

    Ray Roberts
    Participant

    Ted,

    I’m in the same boat as Derek.

    I’m able to restart webhcat from the webhcat server, however I’m unable to do it from Ambari. The webhcat service has no status by its name (i.e. no red or green dot). All other services running on that node (i.e. hdfs, hive, tasktracker) are all green and I can start/stop them just fine.

    I have no pending commands. If I try to start webhcat from the webhcat node view in Ambari, it does nothing and doesn’t show up in the pending commands. If I go to Services view in Ambari, both start and stop is grayed out for webhcat.

    Not sure what is wrong, everything was working a few days ago.

    Collapse
    #23403

    tedr
    Moderator

    Hi Derek,

    Are there still commands pending? Perhaps Ambari got stuck on one of them before the one to stop webhcat.

    Thanks,
    Ted.

    Collapse
    #23320

    Ray Roberts
    Participant

    Ted,

    Thanks for your help,

    Yeah, that does seem to be the logical solution, however I did have the correct FQDN in the command.

    It seems as though something has went wrong within Ambari. In the Ambari UI, both the Start and Stop buttons are greyed out and I noticed a flicker under the Summary section. I can start and stop the webhcat service via root from the webhcat server, so that leads me to believe that nothing is wrong with webhcat itself.

    Any ideas?

    Collapse
    #23315

    tedr
    Moderator

    Hi Ray,

    From your first post it looks like the shell you are using thinks that the $MYHOSTNAME variable was not set when the command was run.

    Thanks,
    Ted.

    Collapse
    #23177

    Ray Roberts
    Participant

    Larry,

    Sorry, I left out some info you requested.

    I’m using Ambari 1.2.2.5 and Chrome & Firefox.

    Thanks,

    Collapse
    #23174

    Ray Roberts
    Participant

    Ted,

    No errors from chrome.

    In /var/log/webhcat/webhcat.log I get:

    ERROR | 24 Apr 2013 09:31:22,242 | org.apache.hadoop.conf.Configuration | Failed to set setXIncludeAware(true) for parser org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@5c1428ea:java.lang.UnsupportedOperation
    Exception: This parser does not support specification “null” version “null”
    java.lang.UnsupportedOperationException: This parser does not support specification “null” version “null”
    at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1123)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1109)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1045)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
    at org.apache.hcatalog.templeton.tool.JobStateTracker.getTrackingJobs(JobStateTracker.java:136)
    at org.apache.hcatalog.templeton.tool.ZooKeeperCleanup.getChildList(ZooKeeperCleanup.java:145)
    at org.apache.hcatalog.templeton.tool.ZooKeeperCleanup.run(ZooKeeperCleanup.java:102)

    In my amber-server.log I get:

    09:49:56,296 ERROR JMXPropertyProvider:292 – Caught exception getting JMX metrics : spec=http://$MYHOSTNAME:50075/jmx
    java.net.NoRouteToHostException: No route to host
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:411)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:525)
    at sun.net.www.http.HttpClient.(HttpClient.java:208)
    at sun.net.www.http.HttpClient.New(HttpClient.java:291)
    at sun.net.www.http.HttpClient.New(HttpClient.java:310)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:987)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:923)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:841)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
    at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:54)
    at org.apache.ambari.server.controller.jmx.JMXPropertyProvider.populateResource(JMXPropertyProvider.java:183)
    at org.apache.ambari.server.controller.jmx.JMXPropertyProvider.populateResources(JMXPropertyProvider.java:118)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl.populateResources(ClusterControllerImpl.java:306)
    at org.apache.ambari.server.controller.internal.Clu

    Collapse
    #23170

    Larry Liu
    Moderator

    Hi, Ray

    What browser you are using to see the flashing red? What version of HDP you are using?

    There are a few things to try:

    1. Go to webhcat page from ambari UI using Chrome and click command-alt-c to start the debug console.
    2. Check if there is an JS error on the page.
    3. If there is no JS error, can you please check ambari server log and ambari agent log.

    From the error you get, it is most likely a JS error.

    Thanks
    Larry

    Collapse
    #23168

    Ray Roberts
    Participant

    Ted,

    I’m having the same issue with WebHCat stuck flashing red……

    I tried what you recommended, however I’m receiving the following:

    # curl -u $MYUSERNAME:$MYPASSWORD -i -X PUT -d ‘{“ServiceInfo”: {“state” : “INSTALLED”}}’ http://$MYHOSTNAME:8080/api/v1/clusters/$MYCLUSTERs/services/WEBHCAT
    curl: (3) [globbing] unmatched brace at pos 13

    curl: (6) Couldn’t resolve host ‘:’
    HTTP/1.1 500 Unable to parse json: org.codehaus.jackson.JsonParseException: Unexpected character (‘’ (code 8216 / 0×2018)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’) at [Source: java.io.StringReader@34d4a37f; line: 1, column: 3]
    Set-Cookie: AMBARISESSIONID=5bzzq69ohbcmnfxreyv2eu8h;Path=/
    Cache-Control: must-revalidate,no-cache,no-store
    Content-Type: text/html;charset=ISO-8859-1
    Content-Length: 1809
    Server: Jetty(7.6.7.v20120910)

    Error 500 Unable to parse json: org.codehaus.jackson.JsonParseException: Unexpected character (‘?’ (code 8216 / 0×2018)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’)
    at [Source: java.io.StringReader@34d4a37f; line: 1, column: 3]

    HTTP ERROR: 500
    Problem accessing /api/v1/clusters/$MYCLUSTERs/services/WEBHCAT. Reason:
    Unable to parse json: org.codehaus.jackson.JsonParseException: Unexpected character (‘?’ (code 8216 / 0×2018)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’)
    at [Source: java.io.StringReader@34d4a37f; line: 1, column: 3]
    Powered by Jetty://

    Collapse
    #21776

    Yi Zhang
    Moderator

    Hi Derek,

    Can you try this api call and see what is the status there?

    http://$webhcat-server-addr:8080/api/v1/clusters/$clustername/services/WEBHCAT

    or access the webhcat service:

    http://$webhcat-server-addr:50111/templeton/v1/status

    Anything in the webhcat logs? Normally it is in /var/log/webhcat/

    Can you manually start webhcat?
    su -l hcat -c “/usr/lib/hcatalog/sbin/webhcat_server.sh start”

    Thanks,
    Yi

    Collapse
    #21774

    Derek Wilson
    Member

    Ted,
    Thanks – That did get the WebHCat added to the blue box (pending commands). I still cannot stop or start the WebHCat from the Services WebHCat screen in Ambari. Still a flashing red light.

    Collapse
    #21739

    tedr
    Moderator

    Hi Derek,

    Thanks for using Hortonworks Data Platform.

    Try this:
    If there are no commands pending for Ambari – you can tell this by looking in the Ambari web UI near the cluster name, if there is a small blue box with a number in it there are pending commands and you must wait till the finish.

    - Open a terminal/console on the the Ambari server node
    - set three environment variables: HOST – the host running the ambari-server, CLUSTER_NAME – the name of your cluster, PASSWORD, the Ambari admin user’s password
    - then issue the following command to stop the service:
    curl –user admin:$PASSWORD -i -X PUT -d ‘{“ServiceInfo”: {“state” : “INSTALLED”}}’ http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/WEBHCAT
    - then to start the service back up again issue the following command:
    curl –user admin:$PASSWORD -i -X PUT -d ‘{“ServiceInfo”: {“state” : “STARTED”}}’ http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/WEBHCAT

    Thanks,
    Ted.

    Collapse
Viewing 23 replies - 1 through 23 (of 23 total)