Home Forums HBase Too many open files — RS logs

This topic contains 3 replies, has 2 voices, and was last updated by  tedr 1 year, 9 months ago.

  • Creator
    Topic
  • #12018

    Laurentiu
    Member

    While scanning the RS log on the XX.XX.XX.105 I noticed the following messages :
    2012-11-12 23:58:57,213 INFO org.apache.hadoop.hdfs.DFSClient: Try reading via the datanode on /XX.XX.XX.105:50010
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.105:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.102:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.104:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_2899571066768819518_24346 from any node: java.io.IOException: No live
    nodes contain current block. Will get new block locations from namenode and retry…
    2012-11-12 23:59:00,214 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_2899571066768819518_24346 from cache because local file /hadoop/hdfs/data/sdf1/current/subdir13/blk_2899571066768819518 could not be opened.
    2012-11-12 23:59:00,214 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_2899571066768819518_24346 on local machinejava.io.FileNotFoundException: /hadoop/hdfs/data/sdf1/current/subdir13/blk_2899571066768819518 (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    What is the explanation for this warning and what can be done to mitigate it ?

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #12067

    tedr
    Member

    Hi Laurentiu,

    Thanks for letting us know.

    Ted

    Collapse
    #12035

    Laurentiu
    Member

    I have found that the tips in http://hbase.apache.org/book.html#trouble.rs.runtime.filehandles very useful. I followed them and fixed my issue by renaming the /etc/security/limits.d/hbase.nofiles.conf to /etc/security/limits.d/hbase.conf and adding this line: hbase – nproc 65536 on all nodes.

    Collapse
    #12031

    tedr
    Member

    Hi Laurentiu,

    The ‘Too many open files’ error is usually returned by the Linux file system. Check to see what your ‘ulimit’ on the box where this error is showing is set to.

    Hope this helps.
    Ted.

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)