Home Forums HDP on Linux – Installation HIVE METASTORE status check failed (HDP 1.2)

This topic contains 30 replies, has 5 voices, and was last updated by  Krishna Gadiraju 11 months ago.

  • Creator
    Topic
  • #14648

    Thomas Emge
    Member

    After my initial HDP deployment with ambari I am encountering issues with the hive service. Where do I start the troubleshooting? I am using CentOS 6.

    Thanks,
    - Thomas

    EXECUTE HIVE_SERVICE_CHECK on c4
    exitcode: 0
    stderr: None
    stdout: notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_1471783679.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_681561435.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException [Error 10001]: Table not found hcatsmokeid2e0a3866_date103013
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: Hive history file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_2005665609.txt
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    err: /Stage[2]/Hdp-hcat::Hcat::Service_check/Exec[/tmp/hcatSmoke.sh]/returns: change from notrun to 0 failed: su – ambari_qa -c ‘sh /tmp/hcatSmoke.sh hcatsmokeid2e0a3866_date103013′ returned 64 instead of one of [0] at /var/lib/ambari-agent/puppet/modules/hdp-hcat/manifests/hcat/service_check.pp:54
    notice: /Stage[2]/Hdp-hcat::Hcat::Service_check/Hdp-hadoop::Exec-hadoop[hcat::service_check::test]/Hdp::Exec[hadoop --config /etc/hadoop/conf fs -test -e /apps/hive/warehouse/hcatsmokeid2e0a3866_date103013]/Anchor[hdp::exec::hadoop –config /etc/hadoop/conf fs -test -e /apps/hive/ware

Viewing 30 replies - 1 through 30 (of 30 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #40789

    Hi Seth!

    Can you please specify the solution for this problem? I am facing the same issue on my 4-machine cluster and have been unable to resolve the problem.

    Collapse
    #15054

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Thanks for letting me know and using HDP.

    Seth

    Collapse
    #15039

    Thomas Emge
    Member

    Seth,

    thanks for your help yesterday. I rebuilt the cluster with the network changes we had talked about yesterday. All the services are now up and running and I have now a properly working Hadoop cluster.

    Thanks,
    - Thomas

    Collapse
    #14977

    Thomas Emge
    Member

    Thanks that would be greatly appreciated,
    - Thomas

    Collapse
    #14976

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Please let us know if you would like us to take a look at your issue over WebEx.

    Thanks,
    Seth

    Collapse
    #14975

    Thomas Emge
    Member

    Seth,

    essentially even with using the Ambari UI to start the services I am back to the metastore and the hiveserver2 service not becoming enabled after the scripts finish.
    I had never really changed the config defaults and had pretty much accepted the Ambari suggested defaults.

    Thanks,
    - Thomas

    Collapse
    #14967

    tedr
    Member

    Hi Thomas,

    If Signing out and back in doesn’t work, try clearing your browser’s cache between logging out and back in. If that still doesn’t work, in your browser’s address field replace every thing after the "#/" with "main"

    Thanks,
    Ted.

    Collapse
    #14963

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Please try to sign out and try again.

    Thanks,
    Seth

    Collapse
    #14962

    Thomas Emge
    Member

    Seth,

    a quick side question….I am attempting to log into the Ambari UI (http://c1:8080/#/login) and as opposed to the dashboard and seeing the installation wizard again. Should that be a reason for concern?

    Thanks,
    - Thomas

    Collapse
    #14961

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Can you please check if you see any processes in /var/run/hive? If yes, can you please try to kill it and restart Hive from Ambari UI. Please let us know if you can find any logs if it still fails to start. You can also go to Ambari UI -> Hive-> Configs and check for any obvious wrong configurations. If all does not work we can try to do a WebEx.

    Thanks,
    Seth

    Collapse
    #14954

    Thomas Emge
    Member

    Ted,

    logging in as the hive user I see get following

    hive> show databases;
    FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    Collapse
    #14941

    tedr
    Member

    Hi Thomas,

    Could you check to see if there was a ‘hive’ database created in MySQL with a “show databases;”?

    Thanks,
    Ted.

    Collapse
    #14925

    Thomas Emge
    Member

    Seth,

    if by UI you mean the Ambari interface I’ve have tried that as well.

    When starting the metastore service I see 2 entries in the dialog like
    Show info about START HIVE_METASTORE on c3
    Show info about START HIVE_SERVICE_CHECK on c4

    After a couple of minutes essentially the first entry is removed from the dialog and the second line turns red with the results shown in the first post.
    - Thomas

    Collapse
    #14924

    Seth Lyubich
    Keymaster

    Thomas,

    Can you please try to start Hive metastore from UI?

    Thanks,
    Seth

    Collapse
    #14923

    Thomas Emge
    Member

    Seth,

    unfortunately that’s what I did but to no success. Maybe I should be more specific, something did happen. After issuing the command to start the metastore service it does come back with an exit code of 1.
    The /var/log/hive/hive.log file then contains the exception stated below and nothing is running at port 9083

    Collapse
    #14920

    Seth Lyubich
    Keymaster

    HI Thomas,

    I think this forum site does some special formatting. Looking on my last post I see that there should be two dashes before ‘service’ in the command I gave you. Can you try to start Hive metastore with two dashes? You can look here for an example:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.0/bk_installing_manually_book/content/rpm-chap6-6.html

    You can also try to start Hive metastore from UI and verify that port is bound to hive proces (see my last comment).

    Please let me know if this helps.

    Seth

    Collapse
    #14919

    Thomas Emge
    Member

    Just to add to the previous post.

    SELinux is off and iptables is stopped.

    Collapse
    #14918

    Thomas Emge
    Member

    Looking at my Ambari dashboard it is telling me that Hive Metastore and Hiveserver2 are not running. If I am attempting to start the service with the command line nothing seems to happen.

    /var/log/hive/hive.log is showing this one:
    javax.jdo.JDOFatalDataStoreException: Communications link failure

    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:407)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:604)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
    ……

    Collapse
    #14915

    Seth Lyubich
    Keymaster

    Thomas,

    Also just to add to Larry’s last comment – can you please make sure that your Hive metastore is started? You can try looking for process on port 9083:

    netstat -anp|grep 9083

    then you can look for process running on that port with ps command.

    If service is not running you can try starting with:

    su – hive -c ‘env HADOOP_HOME=/usr nohup hive –service metastore > /var/log/hive/hive.out 2> /var/log/hive/hive.log & ‘

    Please let me know if this helps.

    Thanks,
    Seth

    Collapse
    #14909

    Larry Liu
    Moderator

    Hi, Thomas

    Can you please do the same test from C3 where your hive metastore service is installed?

    Thanks

    Larry

    Collapse
    #14907

    Thomas Emge
    Member

    Larry,
    I didn’t get too far with hive:

    hive> create table orders (orderid bigint, customerid bigint, productid int, qty int, rate int, estdlvdate string, status string) row format delimited fields terminated by “,”;
    FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

    Collapse
    #14906

    Larry Liu
    Moderator

    Hi Thomas,

    Can you try to run any hive query from your hive client? If you don’t see any issue, it mgiht be some issues during installation smoke test.

    If you still have same issue, please paste log here. We can do web share to triage the issue if you are available.

    Thanks

    Larry

    Collapse
    #14903

    Thomas Emge
    Member

    Seth,

    that connection works fine and it is the one I used to issue the
    mysql> select * from user where user=’hive’;
    selection.

    - Thomas

    Collapse
    #14788

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Did you try to connect to MySql with what I suggested in last comment?

    #mysql -h [servername] -u[username for hive] -p[password]

    I believe in your case it should be something like below from hive client which I believe is c4:

    #mysql -h c3 -u hive -p[password]

    Please let me know what you get.

    Thanks,
    Seth

    Collapse
    #14757

    Thomas Emge
    Member

    Seth,

    yes, I did that, so the permissions to the MySQL instance should be fine. Connectivity should be fine as well, ping and ssh work fine, from c1 (the name node) to c3 (the hive server), from c4 (hive client) to c3 and also from c4 to c3.

    mysql> select * from user where user=’hive’;
    ———-+———————-+
    | Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections |
    +————-+——+——————————————-+————-+————-+————-+————-+————-+———–+————+———————-+
    | c4 | hive | *8B19522D93928599B8F28C88129119684EBD96EB | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |
    | c3 | hive | *8B19522D93928599B8F28C88129119684EBD96EB | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |

    Collapse
    #14730

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Looks like few posts on this thread are missing so I will try to respond:

    1. Make sure that client machine can connect to MySQL. Make sure that basic networking works. You can try pinging the server by IP and hostname.

    2. Make sure that you can connect from client machine:

    #mysql -h [servername] -u[username for hive] -p[password]

    3. Please make sure that you Have all grants setup. You should have ‘Y’ for all privilegies in the user table like below.

    mysql> select * from user where User=’hive’;
    ….
    | Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections |
    +———–+——+——————————————-+————-+————-+————-+————-+————-+———–+————-+—————+————–+———–+————+—————–+————+————+————–+————+———————–+——————+————–+—————–+——————+——————+—————-+———————+——————–+——————+————+————–+———-+————+————-+————–+—————+————-+—————–+———————-+
    | ambari1 | hive | *B34D36DA2C3ADBCCB80926618B9507F5689964B6 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | | | | 0 | 0 | 0 | 0 |
    | localhost | hive | *B34D36DA2C3ADBCCB80926618B9507F5689964B6 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | Y

    Hope this helps,

    Thanks,
    Seth

    Collapse
    #14725

    Thomas Emge
    Member

    Seth,

    I had Ambari create one. After seeing the smoke test fail I started reading through the manual installations steps today and actually granted those privileges for the hive user. However the test is still failing.

    Is there another service that needs to be restarted or something else needs to happen for the test to succeed?

    - Thomas

    Collapse
    #14721

    Seth Lyubich
    Keymaster

    Hi Thomas,

    This looks like an issue with MySQL grants. Did you set up MySQL user yourself, or let Ambari to create one? Make sure that your user have all grants in MySQL. You can test grants by trying to connect to MySql with user that you created for Hive. If it fails you can try to grant privilegies with something like this:

    GRANT ALL PRIVILEGES ON *.* TO ‘dbusername’@’%’;
    flush privileges;

    Hope this helps,

    Thanks,
    Seth

    Collapse
    #14718

    Thomas Emge
    Member

    Seth,

    each machine has 4 GB RAM. The hive server is running on machine c3 and the client is running on machine c4.

    I was suspecting some MySQL issues as well and I did see this message in /var/log/hive/hive.log:

    WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
    javax.jdo.JDOFatalDataStoreException: null, message from server: “Host ‘c3′ is not allowed to connect to this MySQL server”
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:407)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:604)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)

    However I haven’t figured out why c3 couldn’t connect to MySQL running on the same machine. The user and host in the MySQL user table seem to be ok with ‘hive’ as the user and ‘c3′ as the host.

    The mentioned txt file only contains
    SessionStart SESSION_ID=”ambari_qa_201301301411″ TIME=”1359583891158″

    Collapse
    #14658

    Seth Lyubich
    Keymaster

    Hi Thomas,

    Thanks for trying HDP.

    Can you please let me know what environment you are using for your installation, how much memory on your server(s), and how much you setting for Hive?

    The issue appears that metastore cannot be instantiated and smoke test fails. The reasons could be in MySql, credentials, etc. You can try looking in the logs here to see if you can get more details – file=/tmp/ambari_qa/hive_job_log_ambari_qa_201301301411_2005665609.txt.

    Please let me know if this helps.

    Thanks,
    Seth

    Collapse
Viewing 30 replies - 1 through 30 (of 30 total)