Home Forums Hive / HCatalog ORC FileFormat NPE

This topic contains 2 replies, has 1 voice, and was last updated by  Nicolas Maillard 5 months ago.

  • Creator
    Topic
  • #49397

    Nicolas Maillard
    Participant

    Hello

    Testing out ORC I have created a mirror table stored in ORC. On a inner join query my query fails with NPE exception

    the stacktrace
    dMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160)
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“accountid”:null,”visitorid”:null,”sessionid”:null,”sessionnumber”:null,”hostname”:null,”source”:null,”medium”:null,”keyword”:null,”campaign”:null,”content”:null,”gclid”:null,”client_timestamp”:null,”iscookiesenabled”:null,”useragent”:null,”hash”:null,”remoteip”:null,”serverdate”:null,”serverhour”:null,”customvariables”:null,”dt”:”2014-02-20″,”event_type”:”event”}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
    … 8 more
    Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcMapObjectInspector.getMapValueElement(OrcStruct.java:319)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDFIndex.evaluate(GenericUDFIndex.java:102)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:77)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual.evaluate(GenericUDFOPEqual.java:38)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:77)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPOr.evaluate(GenericUDFOPOr.java:53)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:77)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPOr.evaluate(GenericUDFOPOr.java:53)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericF

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #49399

    Nicolas Maillard
    Participant

    I have found the problem sometimes my customvariables map is null
    so adding a customvariables is not null to the query fixes this problem
    I am unclear as to why this is not needed in a normal text format table

    Collapse
    #49398

    Nicolas Maillard
    Participant

    To make things clearer here is the query
    SELECT
    dt as date_event,
    visitorid as visitorid_event,
    sessionid as sessionid_event,
    customvariables["productCategory"] as rayons,
    customvariables["eventCategory"] as eventCategory,
    customvariables["eventAction"] as eventAction,
    CASE WHEN customvariables["productCategory"] like ‘%somthing%’ THEN “something”
    WHEN customvariables["productCategory"] like ‘%somethingelse%’ THEN “somethingelse”
    ELSE “lastthing” END as version_event
    FROM mydb.table_orc mf
    INNER JOIN otherdb.othertable hits
    ON (
    mf.visitorid = hits.visitorid_hit
    AND mf.sessionid = hits.sessionid_hit
    AND mf.dt = hits.date_hit
    )
    WHERE
    event_type = “event”
    AND customvariables["productCategory"] is not null
    AND customvariables["eventCategory"] is not null
    and customvariables["productCustomValue1"] is not null
    AND mf.visitorid is not null
    AND customvariables["eventAction"] is not null
    AND (customvariables["productCategory"] = “fallthings”
    OR customvariables["productCategory"] = “something/allthings”
    OR customvariables["productCategory"] = “something/allthings2″)

    AND ((customvariables["eventCategory"] = “list” AND customvariables["productCustomValue1"] = ‘[{"a":"b","c":"d"}]‘)
    OR customvariables["eventAction"] = “see”
    OR customvariables["eventAction"] = “click”)

    Has anyone come accross this?

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)