Home Forums HDFS Error while writing 1 GB file on hadoop 5 node cluster

Tagged: 

This topic contains 8 replies, has 4 voices, and was last updated by  Robert 1 year ago.

  • Creator
    Topic
  • #32696

    Swapnil Patil
    Participant

    I have hadoop cluster with 5 datanodes and 1 namenode. When i try to write 1 GB file on hadoop cluster.
    $ hadoop jar hadoop-*test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000 (With this command)
    but when I fired this command it gives me following error..

    exception in createblockoutputstream java.net.sockettimeoutexception
    plz help me out of this.

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘Error while writing 1 GB file on hadoop 5 node cluster’ is closed to new replies.

  • Author
    Replies
  • #34150

    Robert
    Participant

    Hi Swapnil,
    Thanks for the solution. To gain more visibility in the forums for this new error you are getting please create a new thread for the error. The too many fetch errors in searching seems to indicate possibly a network issue (bad /etc/hosts file) or bad drive on the slave nodes.

    Regards,
    Robert

    Collapse
    #33638

    Swapnil Patil
    Participant

    I got the solution.. My datanode’s firewall was blocking it from writing blocks..
    Now i have disabled the firewall its working all fine..
    But when I tried to write 50 GB file (with 3 replication) its giving me error
    Too many fetch failures..
    when i analyzed namenode log..its giving me

    org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /benchmarks/TestDFSIO/io_data/test_io_8 for DFSClient_attempt_201308291201_0003_m_000048_1 on client 173.20.1.70, because this file is already being created by DFSClient_attempt_201308291201_0003_m_000048_0 on 173.20.1.73

    Collapse
    #33576

    Robert
    Participant

    Hi Swapnil,
    I saw the following post which may help

    http://computingnote.blogspot.com/2012/04/dfsclient-write-timeout-in-hadoop.html

    It suggest to increase timeouts for both
    dfs.socket.timeout, for read timeout
    dfs.datanode.socket.write.timeout, for write timeout

    Hope that helps.
    Regards,
    Robert

    Collapse
    #33494

    Swapnil Patil
    Participant

    Hi Abdelrahman,
    I am using 100MB switch.. Is that ok? or I need 1 Gbps switch?

    Collapse
    #33478

    Swapnil Patil
    Participant

    Hi Abdelrahman..
    Thanks for your rpl..
    Can u please tell me how can I reduce over commuting of M-R configuration per node.. (I have 1 master and 5 slaves)
    What property should I set for Datanode Handlers..

    Collapse
    #33383

    abdelrahman
    Moderator

    Hi Swapnil,

    It is very possible that it could be one of the following reasons.

    1. Network bottleneck.
    2. Over commuting of M-R configuration per node.
    3. Namenode, xcievers and Datanode Handlers in hdfs-site.xml.

    Please adjust these configurations and try it again.

    Thanks
    -Abderlahman

    Collapse
    #33044

    Swapnil Patil
    Participant

    Hi Seth,
    Thanks for your reply.. Its also giving me an error which says..
    69000 milli secs waiting for the channel to get ready..

    Is that causing because of hardware problem? or network bottleneck?

    Collapse
    #32869

    Seth Lyubich
    Keymaster

    Hi,

    Can you please check that your Datanodes are running? Also, can you please check Namenode log file for any errors?

    Hope this helps,

    Thanks,
    Seth

    Collapse
Viewing 8 replies - 1 through 8 (of 8 total)