Home Forums HDP on Linux – Installation Error installing hive and hcatalog

Tagged: ,

This topic contains 18 replies, has 2 voices, and was last updated by  tedr 1 year, 6 months ago.

  • Creator
    Topic
  • #13214

    Trang Nguyen
    Member

    Hi,

    I’ve tried many times but cannot get Hive to pass the smoke test as part of HMC installation.

    I’ve also manually installed mysql-server and mysql-client on CentOS6. The main issues I’ve had:
    1. mysql-server does not appear to install correctly on centos6.I’ve had to manually install both the server and the client.As a result, it fails the dependency test for hive.
    2. It looks like the default Hive config installed on xdc-tst-mapre-003 does not use mysql, though the installing requires a running instance of mysql:

    hive-site.xml





    javax.jdo.option.ConnectionURL
    jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true
    JDBC connect string for a JDBC metastore

    javax.jdo.option.ConnectionDriverName
    org.apache.derby.jdbc.EmbeddedDriver
    Driver class name for a JDBC metastore

    3. On a different bo (xdc-tst-mapre-004), I tried to manually install hive following your instructions and then ran HDP install.
    However, it also fails with the following error:

    -bash: export: `\u2013service’: not a valid identifier

    I’m not sure how to resolve, so I’ve given up for now unless someone can provide some clues. I’ve also ftp’d the results of the check.sh to your ftp server on Jan 4 (checkHmc_hive_failure.out).

    Thanks,
    Trang

Viewing 18 replies - 1 through 18 (of 18 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #13695

    tedr
    Member

    Hi Trang,

    Will do.

    Thanks,
    Ted.

    Collapse
    #13333

    Trang Nguyen
    Member

    Hi Ted,

    I was out most of the day today. Yes you can contact me directly via e-mail. I could redo the hive installation via HMC if needed.

    Thanks,
    Trang

    Collapse
    #13263

    Trang Nguyen
    Member

    Hi Ted,

    Sure you can contact me directly by e-mail.

    Thanks,
    Trang

    Collapse
    #13261

    tedr
    Member

    Hi Trang,

    Indeed something is a bit unusual here, can we continue this via email? If so I’ll contact you that way.

    Thanks,
    Ted.

    Collapse
    #13258

    Trang Nguyen
    Member

    Hi Ted,

    Both hive and its metastore are on the same host (I didn’t specify a server name during HDP installation). However the issue is that on that server (xdc-tst-mapre-003), it did not correctly configure hive:
    [root@xdc-tst-mapre-003 conf]# cat hive-site.xml





    javax.jdo.option.ConnectionURL
    jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true
    JDBC connect string for a JDBC metastore

    javax.jdo.option.ConnectionDriverName
    org.apache.derby.jdbc.EmbeddedDriver
    Driver class name for a JDBC metastore

    Thanks,
    Trang

    Collapse
    #13254

    tedr
    Member

    Hi Trang,

    If you installed mysql on a host that is different than that which you ar running HMC and have not set MySQL to allow for remote connections HMC will not be able to connect. In this scenario Hive will most probably not be able to connect as well. In your print out of the running process above you can tell that MySQL has not been set for remote connections as the bind address is not set. If all connections to this mysql database are coming from the same host on which it was installed it will allow the connections. To test if it is set to allow remote connections, install mysql client on a host other than the one where mysql-server is installed, then from that box where you just installed the client, try connecting with mysql -u hive -ppassword -h (hostname or ip is that from the mysql-server box).

    Thanks,
    Ted.

    Collapse
    #13252

    Trang Nguyen
    Member

    Hi Ted,
    Here is the running process:
    [root@xdc-tst-mapre-003 lib]# ps -ef|grep mysql
    root 12066 1 0 Jan04 pts/0 00:00:00 /bin/sh /usr/bin/mysqld_safe –datadir=/var/lib/mysql –pid-file=/var/lib/mysql/xdc-tst-mapre-003.openmarket.com.pid
    mysql 12144 12066 0 Jan04 pts/0 00:04:18 /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib64/mysql/plugin –user=mysql –log-error=/var/lib/mysql/xdc-tst-mapre-003.openmarket.com.err –pid-file=/var/lib/mysql/xdc-tst-mapre-003.openmarket.com.pid

    Variables (–variable-name=value)
    and boolean options {FALSE|TRUE} Value (after reading options)
    ————————————————- ————————
    abort-slave-event-count 0
    allow-suspicious-udfs FALSE
    auto-increment-increment 1
    auto-increment-offset 1
    autocommit TRUE
    automatic-sp-privileges TRUE
    back-log 50
    basedir /usr/
    big-tables FALSE
    bind-address (No default value)
    binlog-cache-size 32768

    character-sets-dir /usr/share/mysql/charsets/
    chroot (No default value)
    collation-server latin1_swedish_ci
    completion-type NO_CHAIN
    concurrent-insert AUTO
    connect-timeout 10
    console FALSE
    datadir (No default value)
    date-format %Y-%m-%d
    datetime-format %Y-%m-%d %H:%i:%s
    default-storage-engine InnoDB
    default-time-zone (No default value)

    [root@xdc-tst-mapre-003 lib]# su hive
    bash-4.1$ mysql -u hive -ppassword
    Welcome to the MySQL monitor. Commands end with ; or \g.
    Your MySQL connection id is 8
    Server version: 5.5.24 MySQL Community Server (GPL)

    Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    mysql> show grants;
    +———————————————————————————————————————-+
    | Grants for hive@localhost |
    +———————————————————————————————————————-+
    | GRANT ALL PRIVILEGES ON *.* TO ‘hive’@'localhost’ IDENTIFIED BY PASSWORD ‘*2470C0C

    Collapse
    #13251

    tedr
    Member

    Hi Trang,

    Just remembered something else with manual install of MySQL, it needs to be set to allow remote connections.

    Thanks,
    Ted.

    Collapse
    #13247

    Trang Nguyen
    Member

    Hi Ted,

    I tried it both ways (first without a hive user and then with).

    Thanks,
    Trang

    Collapse
    #13246

    tedr
    Member

    Hi Trang,

    Am I correct in assuming that after you started the mysql service you set up the hive user in mysql before starting HMC?

    Thanks,
    Ted.

    Collapse
    #13234

    Trang Nguyen
    Member

    Hi Bret,

    These are steps I did:

    yum install mysql mysql-server
    yum install MySQL-client
    service mysql start

    Thanks,
    Trang

    Collapse
    #13233

    tedr
    Member

    Hi Trang,

    The file you uploaded has helped me find a possible cause, for some reason when trying to install mysql-server and mysql client HMC could either not connect or could not find the mysql packages in the repositories that you have on the host one which you are running HMC. When you manually installed HMC did you do the install with a ‘yum install mysql-server’ or did you download the package manually from Oracle’s site?

    Thanks,
    Ted.

    Collapse
    #13232

    Trang Nguyen
    Member

    Hi Ted,

    The file that I ftp’d over (checkHmc_hive_failure.out) to your server shows the puppet failure on xdc-tst-mapre-003 which uses the defaults determined by HMC for the hive installation. I manually configured a separate instance of hive on xdc-tst-mapre-004 after the installation on 003 did not work.

    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Anchor[hdp::exec::mysqld start::begin] (notice): Dependency Package[mysql] has failures: true
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Anchor[hdp::exec::mysqld start::begin] (notice): Dependency Package[mysql-server] has failures: true
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Anchor[hdp::exec::mysqld start::begin] (warning): Skipping because of failed dependencies
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Exec[mysqld start] (notice): Dependency Package[mysql] has failures: true
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Exec[mysqld start] (notice): Dependency Package[mysql-server] has failures: true
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Exec[mysqld start] (warning): Skipping because of failed dependencies
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Anchor[hdp::exec::mysqld start::end] (notice): Dependency Package[mysql] has failures: true
    Sat Jan 05 04:46:27 -0500 2013 /Stage[4]/Hdp-mysql::Server/Hdp::Exec[mysqld start]/Anchor[hdp::exec::mysqld start::end] (notice): Dependency Package[mysql-server] has failures:

    Let me know if that helps.
    Thanks,
    Trang

    Collapse
    #13231

    tedr
    Member

    Hi Trang,

    I’m trying to figure out how you can get me the HMC logs from when you first tried to install it with HMC. If they are still around could you send them? I am guessing that they were overwritten when you did the later installs. If you re-try and it fails, could you send us the logs.

    Thanks,
    Ted.

    Collapse
    #13230

    Trang Nguyen
    Member

    Hi Ted,

    Yes I did configure both the username and password. I tried several times, including removing the mysql binaries from the server prior to installation. However, it looks from the puppet logs that the mysql installation did not work and failed dependency checks during the hive installation. At that point, I tried manually to install mysql and hive but couldn’t get HMC to recognize the instance.

    Trang

    Collapse
    #13229

    tedr
    Member

    Hi Trang,

    When you installed MySQL then, did you configure the user and password? If not then HMC could not connect to MySQL to set up the Hive Metastore.

    Thanks,
    Ted.

    Collapse
    #13228

    Trang Nguyen
    Member

    Hi Ted,

    I started with just the defaults ( no preinstalled instance of MySQL) and entered just the username and password, as you mentioned. The MySQL server installation had issues so I manually installed that and started up mysqld processes, re-ran the installation, but it still failed. The files that HMC configured for hive were derby based and not mysql, perhaps because it somehow thought at mysql wasn’t running?

    Thanks,
    Trang

    Collapse
    #13225

    tedr
    Member

    Hi Trang,

    When you were running the HMC installer did you enter a value for the hive server? If so, this is the problem, the value for the Hive server is to be used to point the installer at an already configured install of MySQL. If you leave the field blank then HMC will choose one of the nodes in your cluster and install and configure MySQL on that node. NOTE: you must enter a mysql username and password, but not the server location.

    Thanks,
    Ted.

    Collapse
Viewing 18 replies - 1 through 18 (of 18 total)