HDFS Forum

EOFException reading from Snappy files

  • #29867
    Lane Lawley

    I’m running a job that reads from two input paths containing Snappy-compressed text files. The mapper gets to about 97% and the reducer gets to about 2% when the EOFExceptions start flooding my screen. After enough of them flood in, the job says “completed” and provides stats and counters as if nothing had gone wrong! This is very bizarre. Here is one of the exceptions.

    13/07/23 18:22:49 INFO mapred.JobClient: Task Id : attempt_201307231815_0003_m_000121_2, Status : FAILED
    at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:130)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:100)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:84)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:77)
    at java.io.InputStream.read(InputStream.java:82)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:139)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:530)
    at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

to create new topics or reply. | New User Registration

  • Author
  • #29887

    Hi Lane,

    Other than being snappy compressed is there anything unusual about these files? Is it possible to test if this happens when the job is run against data that is not snappy compressed?


    Lane Lawley


    Sorry for not getting back sooner. My problem was some corruption introduced in an import job. I deleted the offending files and resumed normally. Thank you for the attention.


    Seth Lyubich

    Hi Lane,

    Thanks for letting us know that you resolved your problem.


The topic ‘EOFException reading from Snappy files’ is closed to new replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.