Home Forums Hive / HCatalog Custom File format for HCatalog

This topic contains 3 replies, has 3 voices, and was last updated by  Akki Sharma 6 months, 2 weeks ago.

  • Creator
    Topic
  • #28836

    Hi,

    Newbie question…
    I have my own file format. The files are saved on HDFS. I would like HCatalog to facilitate to read those files by Hive.
    Something like:

    Hive/MapReduce
    |
    HCatalog
    |
    MyFiles

    Where should I start with?
    Is there any sample integration of other File formats which I can use a reference?
    or simply: Is there any documentation or implementation to create a custom StorageHandler and how to use it?

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #30845

    Akki Sharma
    Moderator

    Hi Subroto,

    The closest documentation I could find is “https://cwiki.apache.org/confluence/display/Hive/StorageHandlers”.

    You can also download the Hive code from “http://apache.mirrors.pair.com/hive/hive-0.11.0/”, look at the code in “./src/hcatalog/storage-handlers/hbase” and try to copy the HBase code to write your custom StorageHandler in Hive.

    Best Regards,
    Akki

    Collapse
    #28977

    Hi Ted,

    Is there anyway to develop a custom input-format or SerDe to achieve this?
    I think Hive achieves this using custom input-format and SerDe.
    If it can be achieved by Custom Input-Format; then how can we integrate the InputFormat to HCatalog??

    Cheers,
    Subroto Sanyal

    Collapse
    #28872

    tedr
    Moderator

    Hi Subroto,

    Yo should look into creating a udf for reading/interpretting you file format.

    Thanks,
    Ted.

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)