HDP on Linux – Installation Forum

HIVE METASTORE status check failed (HDP 1.2)

  • #14648
    Thomas Emge
    Member

    After my initial HDP deployment with ambari I am encountering issues with the hive service. Where do I start the troubleshooting? I am using CentOS 6.

    Thanks,
    – Thomas

    EXECUTE HIVE_SERVICE_CHECK on c4
    exitcode: 0
    stderr: None
    stdout: notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_1471783679.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_681561435.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException [Error 10001]: Table not found hcatsmokeid2e0a3866_date103013
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_2005665609.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    err: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: change from notrun to 0 failed: su – ambari_qa -c ‘sh /tmp/hcatSmoke.sh hcatsmokeid2e0a3866_date103013′ returned 64 instead of one of [0] at /var/lib/ambari-agent/puppet/modules/hdp-hcat/manifests/hcat/service_check.pp:54
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Hdp-hadoop::Exec-hadoop[hcat::service_check::test]/Hdp::Exec[hadoop –config /etc/hadoop/conf fs -test -e /apps/hive/warehouse/hcatsmokeid2e0a3866_date103013]/Anchor[hdp::exec::hadoop –config /etc/hadoop/conf fs -test -e /apps/hive/ware

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #14658
    Seth Lyubich
    Moderator

    Hi Thomas,

    Thanks for trying HDP.

    Can you please let me know what environment you are using for your installation, how much memory on your server(s), and how much you setting for Hive?

    The issue appears that metastore cannot be instantiated and smoke test fails. The reasons could be in MySql, credentials, etc. You can try looking in the logs here to see if you can get more details – file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_2005665609.txt.

    Please let me know if this helps.

    Thanks,
    Seth

    #14718
    Thomas Emge
    Member

    Seth,

    each machine has 4 GB RAM. The hive server is running on machine c3 and the client is running on machine c4.

    I was suspecting some MySQL issues as well and I did see this message in /var/log/hive/hive.log:

    WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    javax.jdo.JDOFatalDataStoreException: null, message from server: “Host ‘c3′ is not allowed to connect to this MySQL server”
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:407)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:604)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)

    However I haven’t figured out why c3 couldn’t connect to MySQL running on the same machine. The user and host in the MySQL user table seem to be ok with ‘hive’ as the user and ‘c3′ as the host.

    The mentioned txt file only contains
    SessionStart SESSION_ID=”ambari_qa_201301301411″ TIME=”1359583891158″

    #14721
    Seth Lyubich
    Moderator

    Hi Thomas,

    This looks like an issue with MySQL grants. Did you set up MySQL user yourself, or let Ambari to create one? Make sure that your user have all grants in MySQL. You can test grants by trying to connect to MySql with user that you created for Hive. If it fails you can try to grant privilegies with something like this:

    GRANT ALL PRIVILEGES ON *.* TO ‘dbusername’@’%';
    flush privileges;

    Hope this helps,

    Thanks,
    Seth

    #14725
    Thomas Emge
    Member

    Seth,

    I had Ambari create one. After seeing the smoke test fail I started reading through the manual installations steps today and actually granted those privileges for the hive user. However the test is still failing.

    Is there another service that needs to be restarted or something else needs to happen for the test to succeed?

    – Thomas

    #14730
    Seth Lyubich
    Moderator

    Hi Thomas,

    Looks like few posts on this thread are missing so I will try to respond:

    1. Make sure that client machine can connect to MySQL. Make sure that basic networking works. You can try pinging the server by IP and hostname.

    2. Make sure that you can connect from client machine:

    #mysql -h [servername] -u[username for hive] -p[password]

    3. Please make sure that you Have all grants setup. You should have ‘Y’ for all privilegies in the user table like below.

    mysql> select * from user where User=’hive';
    ….
    | Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections |
    +———–+——+——————————————-+————-+————-+————-+————-+————-+———–+————-+—————+————–+———–+————+—————–+————+————+————–+————+———————–+——————+————–+—————–+——————+——————+—————-+———————+——————–+——————+————+————–+———-+————+————-+————–+—————+————-+—————–+———————-+
    | ambari1 | hive | *B34D36DA2C3ADBCCB80926618B9507F5689964B6 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |
    | localhost | hive | *B34D36DA2C3ADBCCB80926618B9507F5689964B6 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y

    Hope this helps,

    Thanks,
    Seth

    #14757
    Thomas Emge
    Member

    Seth,

    yes, I did that, so the permissions to the MySQL instance should be fine. Connectivity should be fine as well, ping and ssh work fine, from c1 (the name node) to c3 (the hive server), from c4 (hive client) to c3 and also from c4 to c3.

    mysql> select * from user where user=’hive';
    ———-+———————-+
    | Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections |
    +————-+——+——————————————-+————-+————-+————-+————-+————-+———–+————+———————-+
    | c4 | hive | *8B19522D93928599B8F28C88129119684EBD96EB | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |
    | c3 | hive | *8B19522D93928599B8F28C88129119684EBD96EB | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |

    #14788
    Seth Lyubich
    Moderator

    Hi Thomas,

    Did you try to connect to MySql with what I suggested in last comment?

    #mysql -h [servername] -u[username for hive] -p[password]

    I believe in your case it should be something like below from hive client which I believe is c4:

    #mysql -h c3 -u hive -p[password]

    Please let me know what you get.

    Thanks,
    Seth

    #14903
    Thomas Emge
    Member

    Seth,

    that connection works fine and it is the one I used to issue the
    mysql> select * from user where user=’hive’;
    selection.

    – Thomas

    #14906
    Larry Liu
    Moderator

    Hi Thomas,

    Can you try to run any hive query from your hive client? If you don’t see any issue, it mgiht be some issues during installation smoke test.

    If you still have same issue, please paste log here. We can do web share to triage the issue if you are available.

    Thanks

    Larry

    #14907
    Thomas Emge
    Member

    Larry,
    I didn’t get too far with hive:

    hive> create table orders (orderid bigint, customerid bigint, productid int, qty int, rate int, estdlvdate string, status string) row format delimited fields terminated by “,”;
    FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

    #14909
    Larry Liu
    Moderator

    Hi, Thomas

    Can you please do the same test from C3 where your hive metastore service is installed?

    Thanks

    Larry

    #14915
    Seth Lyubich
    Moderator

    Thomas,

    Also just to add to Larry’s last comment – can you please make sure that your Hive metastore is started? You can try looking for process on port 9083:

    netstat -anp|grep 9083

    then you can look for process running on that port with ps command.

    If service is not running you can try starting with:

    su – hive -c ‘env HADOOP_HOME=/usr nohup hive –service metastore > /var/log/hive/hive.out 2> /var/log/hive/hive.log & ‘

    Please let me know if this helps.

    Thanks,
    Seth

    #14918
    Thomas Emge
    Member

    Looking at my Ambari dashboard it is telling me that Hive Metastore and Hiveserver2 are not running. If I am attempting to start the service with the command line nothing seems to happen.

    /var/log/hive/hive.log is showing this one:
    javax.jdo.JDOFatalDataStoreException: Communications link failure

    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:407)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:604)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
    ……

    #14919
    Thomas Emge
    Member

    Just to add to the previous post.

    SELinux is off and iptables is stopped.

    #14920
    Seth Lyubich
    Moderator

    HI Thomas,

    I think this forum site does some special formatting. Looking on my last post I see that there should be two dashes before ‘service’ in the command I gave you. Can you try to start Hive metastore with two dashes? You can look here for an example:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.0/bk_installing_manually_book/content/rpm-chap6-6.html

    You can also try to start Hive metastore from UI and verify that port is bound to hive proces (see my last comment).

    Please let me know if this helps.

    Seth

    #14923
    Thomas Emge
    Member

    Seth,

    unfortunately that’s what I did but to no success. Maybe I should be more specific, something did happen. After issuing the command to start the metastore service it does come back with an exit code of 1.
    The /var/log/hive/hive.log file then contains the exception stated below and nothing is running at port 9083

    #14924
    Seth Lyubich
    Moderator

    Thomas,

    Can you please try to start Hive metastore from UI?

    Thanks,
    Seth

    #14925
    Thomas Emge
    Member

    Seth,

    if by UI you mean the Ambari interface I’ve have tried that as well.

    When starting the metastore service I see 2 entries in the dialog like
    Show info about START HIVE_METASTORE on c3
    Show info about START HIVE_SERVICE_CHECK on c4

    After a couple of minutes essentially the first entry is removed from the dialog and the second line turns red with the results shown in the first post.
    – Thomas

    #14941
    tedr
    Member

    Hi Thomas,

    Could you check to see if there was a ‘hive’ database created in MySQL with a “show databases;”?

    Thanks,
    Ted.

    #14954
    Thomas Emge
    Member

    Ted,

    logging in as the hive user I see get following

    hive> show databases;
    FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    #14961
    Seth Lyubich
    Moderator

    Hi Thomas,

    Can you please check if you see any processes in /var/run/hive? If yes, can you please try to kill it and restart Hive from Ambari UI. Please let us know if you can find any logs if it still fails to start. You can also go to Ambari UI -> Hive-> Configs and check for any obvious wrong configurations. If all does not work we can try to do a WebEx.

    Thanks,
    Seth

    #14962
    Thomas Emge
    Member

    Seth,

    a quick side question….I am attempting to log into the Ambari UI (http://c1:8080/#/login) and as opposed to the dashboard and seeing the installation wizard again. Should that be a reason for concern?

    Thanks,
    – Thomas

    #14963
    Seth Lyubich
    Moderator

    Hi Thomas,

    Please try to sign out and try again.

    Thanks,
    Seth

    #14967
    tedr
    Member

    Hi Thomas,

    If Signing out and back in doesn’t work, try clearing your browser’s cache between logging out and back in. If that still doesn’t work, in your browser’s address field replace every thing after the "#/" with "main"

    Thanks,
    Ted.

    #14975
    Thomas Emge
    Member

    Seth,

    essentially even with using the Ambari UI to start the services I am back to the metastore and the hiveserver2 service not becoming enabled after the scripts finish.
    I had never really changed the config defaults and had pretty much accepted the Ambari suggested defaults.

    Thanks,
    – Thomas

    #14976
    Seth Lyubich
    Moderator

    Hi Thomas,

    Please let us know if you would like us to take a look at your issue over WebEx.

    Thanks,
    Seth

    #14977
    Thomas Emge
    Member

    Thanks that would be greatly appreciated,
    – Thomas

    #15039
    Thomas Emge
    Member

    Seth,

    thanks for your help yesterday. I rebuilt the cluster with the network changes we had talked about yesterday. All the services are now up and running and I have now a properly working Hadoop cluster.

    Thanks,
    – Thomas

    #15054
    Seth Lyubich
    Moderator

    Hi Thomas,

    Thanks for letting me know and using HDP.

    Seth

    #40789

    Hi Seth!

    Can you please specify the solution for this problem? I am facing the same issue on my 4-machine cluster and have been unable to resolve the problem.

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.