Home Forums HBase How To Create Partitions In Hbase Table As Like Hive Table Partitions

This topic contains 1 reply, has 2 voices, and was last updated by  abdelrahman 1 year ago.

  • Creator
    Topic
  • #37769

    Hari Krishna
    Member

    We are planning to migrate from CDH3 to CDH4, as part of this migration we also planning to bring HBASE into out system because it also updates to the data, in CDH3 we are using Hive as warehouse.

    Here we are having the major problem in migration, Hive supports partitions to tables. And our system has many tables in different schemas and some tables has partitions base on date, we have the history of data from last 5 years (365 * 5 partitions exists in some tables).

    We want to achieve the same behavior in HBase also, when I browsed I couldnt find the solution for creating partitions in HBase. Can any one help me in implementing this partition wised table creation in HBase.

    Its really great help if we can find at least some work around solutions also.

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #38241

    abdelrahman
    Moderator

    Hi Har,

    If I may suggest using the Hive/HBase integration. From HBase table you can take its snapshot and populate it into Hive partitioned table using HBase/Hive integration.
    Here is an example:
    1. Create HBase table named ‘test’ from HBase shell:

    create ‘test’,’mstore’

    2. Create two Hive tables. The first table named ‘user’ is in Hive and is where the data is stored in Hive. The second table named ‘user_insert’ maps Hive table to the HBase table ‘test’ that is created in step 1.

    CREATE EXTERNAL TABLE user (
    name string,
    id bigint,
    recipient_id bigint,
    type int
    )
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’
    lines terminated by ‘\n’
    STORED AS TEXTFILE
    LOCATION ‘/user/hive/user';
    — The table will need to be partitioned by dt

    CREATE EXTERNAL TABLE user_insert (
    name string,
    id bigint,
    recipient_id bigint,
    type int
    )
    STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’
    WITH SERDEPROPERTIES (“hbase.columns.mapping” = “:key,mstore:name,mstore:id,mstore:recipient_id,mstore:type”)
    TBLPROPERTIES (“hbase.table.name” = “test”);

    3. Insert data from table user to table user_insert. This will populate HBase table named ‘test’.

    set hbase.zookeeper.quorum=zk1,zk2,zk3 ;
    set hive.zookeeper.client.port=2181;
    set zookeeper.znode.parent=/hbase-unsecure ;

    INSERT OVERWRITE TABLE user_insert SELECT * FROM user;
    — The table will need to be partitioned by dt

    Note:
    In order to have all the required jars without the use having hive do “add jar”, please follow these steps:
    – Create a directory called auxlib
    #mkdir /usr/lib/hive/auxlib

    – Copy all of the following Jars to the previous directory:
    /usr/lib/hive/lib/hive-hbase-handler-0.11.0.1.3.0.0-107.jar;
    /usr/lib/zookeeper/zooNkeeper-3.4.5.1.3.0.0-107.jar ;
    /usr/lib/hbase/hbase-0.94.6.1.3.0.0-107-security.jar;
    /usr/lib/hbase/lib/guava-11.0.2.jar;

    Thanks
    -Rahman

    Collapse
Viewing 1 replies (of 1 total)