Pig Forum

Some pig statement fail at pig- in HDP2.0

  • #43157
    Wang Wei

    We use pig and hcatloader to read data in hive table with two partition: address,age
    It seems that it’s a pig parse issue,the loader statement can pass through the pig statement parser,but the filter can’t pass :
    B = FILTER A BY (address == ‘Shunyi’) AND (age == ’20’);

    After i degrade it to low version pig like pig-,it’s ok,but we meet other issues.
    If it’s a pig issue,when can we get the new version pig to use for hdp2.0?
    If not,how can we avoid it?

to create new topics or reply. | New User Registration

  • Author
  • #44229

    Hi Wang,

    Is it possible for you to post the full pig script to test the use case?


    Wang Wei

    Hi Abdelrahman,
    Thanks for your response. The whole script:
    A = LOAD ‘default.customer’ USING org.apache.hcatalog.pig.HCatLoader();
    B = FILTER A BY (address == ‘Shunyi’) AND (age == ’20’);
    dump B;

    We also try Cloudera CDH5,the issue exist also.

    Wang Wei

    How to edit the comment?
    I want to delete the statement : We also try Cloudera CDH5,the issue exist also.

    Wang Wei

    We have know the reason:
    “register”(case-insensitive) is a key word in pig and the lastest pig(it’s ok for previous pig version) don’t allow the key word as schema’s column name, so if we use it as the schema’s column name,the exception will appear.

    B = FILTER A BY (address == ‘Shunyi’) AND (register == ’20′);

    The script above will make the issue,it’s a wrong usage.

    Thank abdelrahman for your response for my question.(I am not a good asker).

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.