Home Forums Oozie Hive run via oozie

This topic contains 8 replies, has 6 voices, and was last updated by  tedr 9 months, 1 week ago.

  • Creator
    Topic
  • #20530

    Bajeesh TB
    Member

    Hello Everyone,

    I need to run a hive job via oozie. I have tested sample jobs of HDP, that was successful. but when I tried a simple hive job ( Only Creating database) but unfortunately that was unsuccessful.

    Please give me the contents of the files for creating database :

    job.properties,my-hive-default.xml,script.q,workflow.xml

    I am waiting for your reply. :)

    Thanks,
    Bajeesh T.B

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #29642

    tedr
    Moderator

    Hi Faisal,

    The discussion at this link should help you http://stackoverflow.com/questions/15749999/oozie-workflow-hive-table-not-found-but-it-does-exist

    Thanks,
    Ted.

    Collapse
    #29625

    OK here is my situation on sandbox
    i am trying to run hive example as hue user here is my job.properties
    ————-
    nameNode=hdfs://sandbox:8020
    jobTracker=sandbox:50300
    queueName=default
    examplesRoot=examples

    oozie.use.system.libpath=true
    oozie.wf.application.path=${nameNode}/user/hue/${examplesRoot}/apps/hive
    ————–
    here is workflow.xml
    ————————

    ${jobTracker}
    ${nameNode}

    mapred.job.queue.name
    ${queueName}

    script.q
    INPUT=/user/hue/${examplesRoot}/input-data-hue2/table
    OUTPUT=/user/hue/${examplesRoot}/output-data-hue2/hive

    Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]

    ————————
    here is my script.q
    ————————–
    CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION ‘${INPUT}’;
    INSERT OVERWRITE DIRECTORY ‘${OUTPUT}’ SELECT * FROM sample_07;
    ————————–
    and here the error i get
    ————————-
    Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10001]
    ————————
    All above files are placed on HDFS /user/hue/examples/apps/hive
    here is output of “[root@sandbox /]# hadoop dfs -ls /apps/hive/warehouse” command
    ——————
    Found 2 items
    drwxrwxrwx – sample hdfs 0 2013-06-10 14:38 /apps/hive/warehouse/sample_07
    drwxrwxrwx – sample hdfs 0 2013-06-10 14:38 /apps/hive/warehouse/sample_08
    ——————
    I am running oozie job as hue user. can any one please help me where i am doing mistake…?

    Collapse
    #29427

    tedr
    Moderator

    Hi Sanjay,

    Thanks for the example!

    Ted.

    Collapse
    #29018

    Hi

    I can help u with this and make sure it works for u

    Here is a Hive Action from my production Oozie workflow…I am using Hue 2.1.0 to create Oozie flows
    NOTE: that this script file is on HDFS
    /workflows/impressions/hivescripts/stats_prodimpr_gz_files.hql

    ${jobTracker}
    ${nameNode}

    oozie.hive.defaults
    /workflows/impressions/hive-site.xml

    hive.exec.compress.output
    true

    mapreduce.output.fileoutputformat.compress.codec
    com.hadoop.compression.lzo.LzopCodec

    mapreduce.output.fileoutputformat.compress
    true

    mapreduce.job.maps
    500

    mapreduce.job.reduces
    8

    mapreduce.tasktracker.map.tasks.maximum
    12

    mapreduce.tasktracker.reduce.tasks.maximum
    8

    mapreduce.map.output.compress
    true

    mapreduce.map.output.compress.codec
    org.apache.hadoop.io.compress.SnappyCodec

    hive.exec.compress.intermediate
    true

    /workflows/impressions/hivescripts/stats_prodimpr_gz_files.hql
    /workflows/impressions/lib/impressions-hiveudfs-1.0.jar

    Contents of stats_prodimpr_gz_files.hql
    ==============
    ALTER TABLE impressions_hive_stats DROP IF EXISTS PARTITION (table_name_partition=’${PRODIMPR_GZ_FILES_HDFS_DIR_NAME}’, aggregation_date_partition=’${DATE_STR}’, metric_name_partition=’record_counts’, metric_key_partition=’all_servers’)
    ;

    Collapse
    #26906

    tedr
    Moderator

    Hi Yogesh,

    On an HDP cluster these examples can be found in /usr/share/doc/examples

    Thanks,
    Ted.

    Collapse
    #26843

    yogesh hd
    Member

    Hi

    I too need a sample script. Could you please share job.properties,my-hive-default.xml,script.q,workflow.xml

    Collapse
    #20827

    Bajeesh TB
    Member

    Hi Seth,

    Thanks for your reply. I will check this and let you know the status.

    Thanks,
    Bajeesh T.B

    Collapse
    #20571

    Seth Lyubich
    Keymaster

    Hi Bajeesh,

    In HDP cluster files that you asked for are located in /usr/share/doc/oozie-3.2.0.21/examples/apps/hive . Hive example actually runs simple commands in script.q file:

    CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION ‘${INPUT}’;
    INSERT OVERWRITE DIRECTORY ‘${OUTPUT}’ SELECT * FROM test;

    Some Hive DDL commands can be found here:

    https://cwiki.apache.org/Hive/languagemanual-ddl.html

    Hope this helps,
    Seth

    Collapse
Viewing 8 replies - 1 through 8 (of 8 total)