Some pig statement fail at pig-0.12.0.2.0.6.0-76-withouthadoop.jar in HDP2.0

to create new topics or reply. | New User Registration

Tagged: 

This topic contains 4 replies, has 2 voices, and was last updated by  Wang Wei 1 year, 3 months ago.

  • Creator
    Topic
  • #43157

    Wang Wei
    Participant

    We use pig and hcatloader to read data in hive table with two partition: address,age
    It seems that it’s a pig parse issue,the loader statement can pass through the pig statement parser,but the filter can’t pass :
    B = FILTER A BY (address == ‘Shunyi’) AND (age == ’20’);

    After i degrade it to low version pig like pig-0.11.2.2.0.5.0-67-withouthadoop.jar,it’s ok,but we meet other issues.
    So
    If it’s a pig issue,when can we get the new version pig to use for hdp2.0?
    If not,how can we avoid it?

Viewing 4 replies - 1 through 4 (of 4 total)

You must be to reply to this topic. | Create Account

  • Author
    Replies
  • #50270

    Wang Wei
    Participant

    We have know the reason:
    “register”(case-insensitive) is a key word in pig and the lastest pig(it’s ok for previous pig version) don’t allow the key word as schema’s column name, so if we use it as the schema’s column name,the exception will appear.

    B = FILTER A BY (address == ‘Shunyi’) AND (register == ’20′);

    The script above will make the issue,it’s a wrong usage.

    Thank abdelrahman for your response for my question.(I am not a good asker).

    Collapse
    #50261

    Wang Wei
    Participant

    How to edit the comment?
    I want to delete the statement : We also try Cloudera CDH5,the issue exist also.

    Collapse
    #50173

    Wang Wei
    Participant

    Hi Abdelrahman,
    Thanks for your response. The whole script:
    A = LOAD ‘default.customer’ USING org.apache.hcatalog.pig.HCatLoader();
    B = FILTER A BY (address == ‘Shunyi’) AND (age == ’20’);
    dump B;

    We also try Cloudera CDH5,the issue exist also.

    Collapse
    #44229

    abdelrahman
    Moderator

    Hi Wang,

    Is it possible for you to post the full pig script to test the use case?

    Thanks
    -Abdelrahman

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.