Home Forums Sqoop Sqoop using LZO compression (YARN)

This topic contains 3 replies, has 2 voices, and was last updated by  Robert Molina 6 months, 3 weeks ago.

  • Creator
    Topic
  • #45390

    Chiew Yea Kuek
    Participant

    Hi All,

    Previously I tested to sqoop data into hive using LZO compression (Hadoop 1), it will create the LZO index file automatically. When I tested to sqoop data into hive using Hadoop 2 (YARN), it will hit error as below when creating the LZO index file.

    2013-12-10 02:28:32,246 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://sandbox:8020/user/oozie/temp/RawDimAgent/part-m-00000.lzo:0+258561
    2013-12-10 02:28:32,267 INFO [main] com.hadoop.mapreduce.LzoSplitRecordReader: Closing input stream after reading 0 blocks from hdfs://sandbox:8020/user/oozie/temp/RawDimAgent/part-m-00000.lzo
    2013-12-10 02:28:32,268 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@32c26ede
    java.lang.NullPointerException
    at com.hadoop.mapreduce.LzoSplitRecordReader.close(LzoSplitRecordReader.java:127)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:499)
    at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1982)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:772)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
    2013-12-10 02:28:32,272 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
    at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:47)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:524)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

    Anyone experience this? Appreciate your input. Thanks a lot !

    Regards,
    KCYea

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Sqoop using LZO compression (YARN)’ is closed to new replies.

  • Author
    Replies
  • #46481

    Robert Molina
    Moderator

    Hi Cyea,
    Can you run a regular mapreduce job with lzo compression?

    Regards,
    Robert

    Collapse
    #45516

    Chiew Yea Kuek
    Participant

    Hi Robert,

    I am using the LZO that pre-installed in the hortonworks sandbox. It is hadoop-lzo-0.5.0.jar, so I believe it is version 0.5. Do you have any clue on this?

    Regards,
    CYea

    Collapse
    #45489

    Robert Molina
    Moderator

    Hi Chiew,
    What version of LZO are you using?

    Regards,
    Robert

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)