Home Forums HDFS EOFException reading from Snappy files

This topic contains 3 replies, has 3 voices, and was last updated by  Seth Lyubich 8 months, 3 weeks ago.

  • Creator
    Topic
  • #29867

    Lane Lawley
    Member

    I’m running a job that reads from two input paths containing Snappy-compressed text files. The mapper gets to about 97% and the reducer gets to about 2% when the EOFExceptions start flooding my screen. After enough of them flood in, the job says “completed” and provides stats and counters as if nothing had gone wrong! This is very bizarre. Here is one of the exceptions.

    13/07/23 18:22:49 INFO mapred.JobClient: Task Id : attempt_201307231815_0003_m_000121_2, Status : FAILED
    java.io.EOFException
    at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:130)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:100)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:84)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:77)
    at java.io.InputStream.read(InputStream.java:82)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:139)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:530)
    at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘EOFException reading from Snappy files’ is closed to new replies.

  • Author
    Replies
  • #30530

    Seth Lyubich
    Keymaster

    Hi Lane,

    Thanks for letting us know that you resolved your problem.

    Thanks,
    Seth

    Collapse
    #30508

    Lane Lawley
    Member

    Ted,

    Sorry for not getting back sooner. My problem was some corruption introduced in an import job. I deleted the offending files and resumed normally. Thank you for the attention.

    Lane

    Collapse
    #29887

    tedr
    Moderator

    Hi Lane,

    Other than being snappy compressed is there anything unusual about these files? Is it possible to test if this happens when the job is run against data that is not snappy compressed?

    Thanks,
    Ted.

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)