Home Forums HDP on Linux – Installation hmc console not reporting service results

This topic contains 21 replies, has 2 voices, and was last updated by  James Solderitsch 2 years, 1 month ago.

  • Creator
    Topic
  • #7649

    This just started today. I can use the manage service interface to kick off various aspects of the installation process, and the pop dialog shows that what I choose to do has started, but this dialog that is showing the various progress bars NEVER refreshes. It seems like there is no communication of completed results.

    I was trying to move off of using localhost.localdomain and used the host file to declare a FQDN — reinstalled puppet and hmc to get a new pem file but now there is no communication between what I kick off and the web page that is supposed to report problems.

    Any suggestions about where to look?

    Jim

Viewing 21 replies - 1 through 21 (of 21 total)

The topic ‘hmc console not reporting service results’ is closed to new replies.

  • Author
    Replies
  • #7772

    No — no firewalls, no SELINUX.

    Collapse
    #7771

    Sasha J
    Moderator

    If you did everything correctly, HMC installs MySQL for you.
    Any firewalling/SELinux running?

    Sasha

    Collapse
    #7756

    The mysql that I am using was the one that Hortonworks obtained during setup. I did not do any special provisioning or account setup. Hortonworks implied that this was all done automatically behind the scenes.

    Did I get the wrong impression?

    Specifically, what extra things do I need to do to supply the added permissions? Is there a recommended section in the Mysql docs to read? What permissions must be added manually?

    Thanks

    Jim

    Collapse
    #7747

    Sasha J
    Moderator

    This error message means that MySQL user does not have needed permissions.

    Please, check documentation on setting up Mysql Account.

    Thank you!
    Sasha

    Collapse
    #7709

    Last posting for me today — did a mouse over the Desciption cell and the full error message pops up:

    CRITICAL: Error accessing hive-metaserver status [Exception in thread "main" java.io.IOException: Permission denied]

    Maybe this gives support something to hang on to in their investigation? What permission is missing?

    Collapse
    #7708

    I am now past the cluster startup — I think my recent errors were because mysqld was in fact not running. I made sure mysql was installed and mysqld running when I did another cluster install. This worked.

    BUT I still see the HIVE-METASTORE status check critical error and no amount of refreshing the Monitoring console makes this error disappear. So apparently the problem wasn’t the MySQL host entry after all.

    As root, I was able to enter the grant instructions Sasha mentioned earlier — for good measure I did one for localhost, jjscentos64 (my hosts file host name) and jjscentos64.local (the FQDN). But these don’e affect the error message either.

    One odd thing is that value in the Duration column is: 0day 7hr 34min but the the cluster hasn’t been up that long. Could this be stale data that the console is bringing in?

    So I am kind of at the same spot as I was in when today began — *sigh*

    Collapse
    #7705

    Once again I started over — I have a high tolerance for installer pain.

    I was worried that I may have a bad version of mysql installed so I did a yum erase of it as well.

    When I did the setup cluster sequence it got to the same failure point as before (Hive/HCatalog test) with what appears to be the same error:

    “\”Fri Jul 27 16:00:16 -0400 2012 /Stage[2]/Hdp-hive::Hive::Service_check/Exec[/tmp/hiveSmoke.sh]/returns (notice): Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided)\””,

    But this time it appears that mysql wasn’t even installed at all. I thought that the cluster install would install it if it was not found. mysqld is an unknown service and mysql is not in the root user’s path.

    Is this what should have happened?

    Still looking for answers.

    Jim

    Collapse
    #7696

    And just for kicks, I logged into mysql as my admin user set up during the install process and I can get the mysql prompt. I tried:

    mysql> grant all privileges on *.* to ‘HCAT_USER’@’jjscentos64.local’ identified by ‘HCAT_PW’;
    ERROR 1045 (28000): Access denied for user ‘hdwDBadmin’@’localhost’ (using password: YES)

    So that does not seem to be an answer for my dilemma either.

    Jim

    Collapse
    #7682

    Further info: in the /var/log/hive file I see messages like this:

    ** BEGIN NESTED EXCEPTION **

    java.net.ConnectException
    MESSAGE: Connection refused

    STACKTRACE:

    java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.(Socket.java:375)
    at java.net.Socket.(Socket.java:218)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
    at com.mysql.jdbc.MysqlIO.(MysqlIO.java:271)
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:2771)
    at com.mysql.jdbc.Connection.(Connection.java:1555)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
    at java.sql.DriverManager.getConnection(DriverManager.java:582)
    at java.sql.DriverManager.getConnection(DriverManager.java:154)
    at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:444)
    at org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:264)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:681)
    at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:290)
    at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:410)
    at org.datanucleus.NucleusContext.initialise(NucleusContext.java:280)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:591)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:346)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:333)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:371)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:278)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:248)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:3017)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:2957)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    ** END NESTED EXCEPTION **

    Does this help diagnose my misconfiguration?

    Collapse
    #7681

    I did some checking after the failed cluster install with mysqld running and it turns out that this command works:

    mysql -h localhost -u hdwDBadmin -p

    But this command does NOT:

    mysql -h jjscentos64.local -u hdwDBadmin -p

    The hostname jjscentos64.local is the FQDN for the experimental setup. I am told that:
    ERROR 1130 (00000): Host ‘jjscentos64.local’ is not allowed to connect to this MySQL server

    Does this help with a work-around? I do NOT want the critical Hive metadata error to be reported AND I want to install a cluster without using localhost.localdomain as my FQDN.

    Maybe this is impossible for the current release?

    Jim

    Collapse
    #7680

    It looks like if I leave the mysql host name undefined, the setup process uses the name localhost. Then Hive installs BUT then the critical error I reported earlier in this discussion happens. If I try to use the FQDN (or the hostname by itself) for the mysql host name, then the cluster install fails. Could it be that there is a bug in the initialization here — I shouldn’t have to muck with mysql grant statements.

    Stuck for now.

    Collapse
    #7679

    I did replace the mysql host with both the FQDN and the assigned hostname. I then reinstalled the cluster and the install FAILS during the Hive test. I have the log file but http://ftp.support.hortonworks.com appears to be unreachable for me right now so I can’t upload the log file from the Hive test failure.

    When I ping the FQDN I do NOT get 127.0.0.1 — I get the address I have in my hosts file: 192.168.150.130.

    This is getting very weird!

    Can you confirm if the ftp site is up?

    Collapse
    #7677

    Sasha J
    Moderator

    if it returns 127.0.0.1

    then you are resolving localhost, but the metastore has put in a grant for your FQDN

    you can simply add a grant in the mysql server to fix this

    grant all privileges on *.* to ‘HCAT_USER’@’localhost’ identified by ‘HCAT_PW’

    Collapse
    #7676

    Sasha J
    Moderator

    what is IP when you ping your HMC server from the HMC server using the FQDN?

    Collapse
    #7670

    Could this issue be due to the choice of Hive hostname? I left this blank and it appears that localhost is then used as the name. Since every other part of the hadoop config is now using the new FQDN, perhaps I should manually enter that name for the mysql host.

    I am trying that now — will report back when the cluster installation completes.

    Collapse
    #7669

    I ftp’d the file just now — named hive-fail.txt. Hopefully support can see it and make some sense of it.

    Collapse
    #7668

    After seeing the error, I tried to stop Hive and this also required stopping Templeton. Templeton stopped OK, but after a bit if waiting, the Hive stop FAILed. Despite the failure, I was able to start Hive and Templeton successfully BUT the same critical error was reported in the hmc console.

    I will upload the operations log documenting the Hive stop failure tomorrow to the Hortonworks website.

    Collapse
    #7667

    Sasha,

    Not sure I know what you are asking. The HMC host is the ONLY host — one VM to rule them all.

    I have hostname: jjscentos64 and I have set the domain name as local so the FQDN is jjscentos64.local.

    This seems to be working for the most part.

    BUT… (there is always a but)

    I am seeing a new critical error related to HIVE-METASTORE that has appeared several times now.

    The alert name is: HIVE-METASTORE status check

    It’s status is CRIT

    The description is: CRITICAL: Error accessing hive-metaserver status [Exception in thread "main"]

    This error did not seem to occur when using localhost.localdomain for the VM.

    I see no other errors. I have not suspended the VM — this is just running after the installation.

    Appreciate suggestions.

    Jim

    Collapse
    #7662

    Sasha J
    Moderator

    Hi James,

    are you certain you are able to resolve all the hosts from the HMC host

    AND

    you are able to resolve the HMC host from all the hosts?

    -Sasha

    Collapse
    #7658

    I was connected, but the dialog reporting progress after each step was not refreshing in Firefox.

    But then I noticed that the results of hostname and hostname -f were different. I had added an entry in /etc/hosts but I forgot about the hostname property in /etc/sysconfig/network which was not set to the FQDN. It was set to the hostname only (e.g. jjscentos64). Once I changed to the FQDN, the browser behaved as it did before.

    Learning more about networking everyday as I go through this exercise.

    Collapse
    #7655

    Sasha J
    Moderator

    did you do a basic check for open ports when you were unable to connect?

    Collapse
Viewing 21 replies - 1 through 21 (of 21 total)