Adding new data volumes

to create new topics or reply. | New User Registration

This topic contains 1 reply, has 2 voices, and was last updated by  tedr 1 year, 10 months ago.

  • Creator
  • #25150


    I’m trying to add some disk space to one of our nodes. We currently have a heterogeneous setup with one data directory configured on each node. I was able to aquire some extra disks for one of the nodes, arranged in a JBOD format. I’m not able to get any for our other nodes at the moment.

    As I understand it, Hadoop should support heterogeneous disk layouts. I understand it’s not optimal, but it’s the best I can do for now.

    The official Hadoop documentation for version 1.1.2 states that the variable “Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.”

    Since diretories that don’t exist should be ignored, I decided to change to the existing directory plus the four new ones, plus five that don’t exist yet for future expansion. I did this through the REST API since it’s not possible to change via Ambari directly. I also changed the “DataNode volumes failure toleration” to 9 so that I wouldn’t get startup errors.

    The problem is that now I get the following error on all the datanodes, which fail to start:

    2013-05-10 13:48:57,650 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid value for validVolsRequired : -8 , Current valid volumes: 1
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(

    Is it even possible to add additional data volumes to a single node? What would be the recommended solution to this, assuming that I cannot acquire extra hard drives for some time?

    Thank you,

Viewing 1 replies (of 1 total)

The topic ‘Adding new data volumes’ is closed to new replies.

  • Author
  • #26433


    Hi John,

    Unfortunately, Ambari does not currently support differing volumes on data nodes, this feature is planeed for a future release.


Viewing 1 replies (of 1 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.