EOFException reading from Snappy files

to create new topics or reply. | New User Registration

This topic contains 3 replies, has 3 voices, and was last updated by  Seth Lyubich 2 years ago.

  • Creator
  • #29867

    Lane Lawley

    I’m running a job that reads from two input paths containing Snappy-compressed text files. The mapper gets to about 97% and the reducer gets to about 2% when the EOFExceptions start flooding my screen. After enough of them flood in, the job says “completed” and provides stats and counters as if nothing had gone wrong! This is very bizarre. Here is one of the exceptions.

    13/07/23 18:22:49 INFO mapred.JobClient: Task Id : attempt_201307231815_0003_m_000121_2, Status : FAILED
    at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:130)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:100)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:84)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:77)
    at java.io.InputStream.read(InputStream.java:82)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:139)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:530)
    at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘EOFException reading from Snappy files’ is closed to new replies.

  • Author
  • #30530

    Seth Lyubich

    Hi Lane,

    Thanks for letting us know that you resolved your problem.



    Lane Lawley


    Sorry for not getting back sooner. My problem was some corruption introduced in an import job. I deleted the offending files and resumed normally. Thank you for the attention.




    Hi Lane,

    Other than being snappy compressed is there anything unusual about these files? Is it possible to test if this happens when the job is run against data that is not snappy compressed?


Viewing 3 replies - 1 through 3 (of 3 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.