Too many open files — RS logs

to create new topics or reply. | New User Registration

This topic contains 3 replies, has 2 voices, and was last updated by  tedr 2 years, 4 months ago.

  • Creator
    Topic
  • #12018

    Laurentiu
    Member

    While scanning the RS log on the XX.XX.XX.105 I noticed the following messages :
    2012-11-12 23:58:57,213 INFO org.apache.hadoop.hdfs.DFSClient: Try reading via the datanode on /XX.XX.XX.105:50010
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.105:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.102:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /XX.XX.XX.104:50010, add to deadNodes and continuejava.net.SocketException:
    Too many open files
    2012-11-12 23:58:57,213 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_2899571066768819518_24346 from any node: java.io.IOException: No live
    nodes contain current block. Will get new block locations from namenode and retry…
    2012-11-12 23:59:00,214 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_2899571066768819518_24346 from cache because local file /hadoop/hdfs/data/sdf1/current/subdir13/blk_2899571066768819518 could not be opened.
    2012-11-12 23:59:00,214 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_2899571066768819518_24346 on local machinejava.io.FileNotFoundException: /hadoop/hdfs/data/sdf1/current/subdir13/blk_2899571066768819518 (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    What is the explanation for this warning and what can be done to mitigate it ?

Viewing 3 replies - 1 through 3 (of 3 total)

You must be to reply to this topic. | Create Account

  • Author
    Replies
  • #12067

    tedr
    Member

    Hi Laurentiu,

    Thanks for letting us know.

    Ted

    Collapse
    #12035

    Laurentiu
    Member

    I have found that the tips in http://hbase.apache.org/book.html#trouble.rs.runtime.filehandles very useful. I followed them and fixed my issue by renaming the /etc/security/limits.d/hbase.nofiles.conf to /etc/security/limits.d/hbase.conf and adding this line: hbase – nproc 65536 on all nodes.

    Collapse
    #12031

    tedr
    Member

    Hi Laurentiu,

    The ‘Too many open files’ error is usually returned by the Linux file system. Check to see what your ‘ulimit’ on the box where this error is showing is set to.

    Hope this helps.
    Ted.

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.