The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

HDFS Forum

HDFS Corrupt after new install.

  • #54183
    Brent DiNicola
    Participant

    Summary: I installed 2.1, then I went to do other things which I have to attend to. The cluster sat idle with nothing happening (It’s not yet being used, this was a prep load) for about 12 days. I have now come back and HDFS is complaining in Ambari so I ran a few checks. Below is the information I have. I don’t understand how these things got corrupted with nothing happening in the cluster except sitting idle. It is concerning that it got corrupted on a new install with nothing happening. Ideas?

    Reported Corrupt Files
    blk_1073742077 /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    blk_1073742204 /user/ambari-qa/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS
    blk_1073742200 /user/ambari-qa/examples/apps/streaming/job.windows.properties
    blk_1073741873 /user/oozie/share/lib/hcatalog/hive-serde-0.13.0.2.1.1.0-385.jar
    blk_1073742172 /user/ambari-qa/examples/apps/hive/workflow.xml
    blk_1073742039 /user/oozie/share/lib/pig/hive-ant-0.13.0.2.1.1.0-385.jar
    blk_1073742028 /user/oozie/share/lib/pig/commons-net-1.4.1.jar
    blk_1073741874 /user/oozie/share/lib/hcatalog/hive-webhcat-java-client-0.13.0.2.1.1.0-385.jar

    Total: At least 8 corrupt file(s)

    [hdfs@server ~]$ hadoop fs -ls /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    -rwxr-xr-x 3 oozie hdfs 9711 2014-05-08 15:28 /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    [hdfs@server ~]$ hadoop fsck /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar -locations -blocks -files
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.

    Connecting to namenode via http://server:50070
    FSCK started by hdfs (auth:SIMPLE) from /172.17.32.20 for path /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar at Tue May 20 17:36:59 CDT 2014
    /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar 9711 bytes, 1 block(s):
    /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar: CORRUPT blockpool BP-1741485632-172.17.32.20-1399580717335 block blk_1073742077
    MISSING 1 blocks of total size 9711 B
    0. BP-1741485632-172.17.32.20-1399580717335:blk_1073742077_1253 len=9711 MISSING!

    Status: CORRUPT
    Total size: 9711 B
    Total dirs: 0
    Total files: 1
    Total symlinks: 0
    Total blocks (validated): 1 (avg. block size 9711 B)
    ********************************
    CORRUPT FILES: 1
    MISSING BLOCKS: 1
    MISSING SIZE: 9711 B
    CORRUPT BLOCKS: 1
    ********************************
    Minimally replicated blocks: 0 (0.0 %)
    Over-replicated blocks: 0 (0.0 %)
    Under-replicated blocks: 0 (0.0 %)
    Mis-replicated blocks: 0 (0.0 %)
    Default replication factor: 3
    Average block replication: 0.0
    Corrupt blocks: 1
    Missing replicas: 0
    Number of data-nodes: 36
    Number of racks: 1
    FSCK ended at Tue May 20 17:36:59 CDT 2014 in 0 milliseconds

    The filesystem under path ‘/user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar’ is CORRUPT

The forum ‘HDFS’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.