Home Forums HDFS HDFS Corrupt after new install.

This topic contains 0 replies, has 1 voice, and was last updated by  Brent DiNicola 2 months ago.

  • Creator
    Topic
  • #54183

    Brent DiNicola
    Participant

    Summary: I installed 2.1, then I went to do other things which I have to attend to. The cluster sat idle with nothing happening (It’s not yet being used, this was a prep load) for about 12 days. I have now come back and HDFS is complaining in Ambari so I ran a few checks. Below is the information I have. I don’t understand how these things got corrupted with nothing happening in the cluster except sitting idle. It is concerning that it got corrupted on a new install with nothing happening. Ideas?

    Reported Corrupt Files
    blk_1073742077 /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    blk_1073742204 /user/ambari-qa/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS
    blk_1073742200 /user/ambari-qa/examples/apps/streaming/job.windows.properties
    blk_1073741873 /user/oozie/share/lib/hcatalog/hive-serde-0.13.0.2.1.1.0-385.jar
    blk_1073742172 /user/ambari-qa/examples/apps/hive/workflow.xml
    blk_1073742039 /user/oozie/share/lib/pig/hive-ant-0.13.0.2.1.1.0-385.jar
    blk_1073742028 /user/oozie/share/lib/pig/commons-net-1.4.1.jar
    blk_1073741874 /user/oozie/share/lib/hcatalog/hive-webhcat-java-client-0.13.0.2.1.1.0-385.jar

    Total: At least 8 corrupt file(s)

    [hdfs@server ~]$ hadoop fs -ls /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    -rwxr-xr-x 3 oozie hdfs 9711 2014-05-08 15:28 /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar
    [hdfs@server ~]$ hadoop fsck /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar -locations -blocks -files
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.

    Connecting to namenode via http://server:50070
    FSCK started by hdfs (auth:SIMPLE) from /172.17.32.20 for path /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar at Tue May 20 17:36:59 CDT 2014
    /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar 9711 bytes, 1 block(s):
    /user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar: CORRUPT blockpool BP-1741485632-172.17.32.20-1399580717335 block blk_1073742077
    MISSING 1 blocks of total size 9711 B
    0. BP-1741485632-172.17.32.20-1399580717335:blk_1073742077_1253 len=9711 MISSING!

    Status: CORRUPT
    Total size: 9711 B
    Total dirs: 0
    Total files: 1
    Total symlinks: 0
    Total blocks (validated): 1 (avg. block size 9711 B)
    ********************************
    CORRUPT FILES: 1
    MISSING BLOCKS: 1
    MISSING SIZE: 9711 B
    CORRUPT BLOCKS: 1
    ********************************
    Minimally replicated blocks: 0 (0.0 %)
    Over-replicated blocks: 0 (0.0 %)
    Under-replicated blocks: 0 (0.0 %)
    Mis-replicated blocks: 0 (0.0 %)
    Default replication factor: 3
    Average block replication: 0.0
    Corrupt blocks: 1
    Missing replicas: 0
    Number of data-nodes: 36
    Number of racks: 1
    FSCK ended at Tue May 20 17:36:59 CDT 2014 in 0 milliseconds

    The filesystem under path ‘/user/oozie/share/lib/pig/slf4j-log4j12-1.6.6.jar’ is CORRUPT

You must be logged in to reply to this topic.