Using the command line to manage files on HDFS

In this tutorial we will walk through some of the basic HDFS commands you will need to manage files on HDFS. To complete this tutorial you will need a working HDP cluster. The easiest way to have a Hadoop cluster is to download the Hortonworks Sandbox.

Let’s get started.

Step 1: Let’s create a directory in HDFS, upload a file and list.

Let’s look at the syntax first:

hadoop fs -mkdir:
  • It will take path uri’s as argument and creates directory or directories.
            hadoop fs -mkdir <paths> 
            hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
            hadoop fs -mkdir hdfs://
hadoop fs -ls:
  • Lists the contents of a directory
  • For a file returns stats of a file
            hadoop fs -ls <args>
            hadoop fs -ls /user/hadoop/dir1 /user/hadoop/dir2
            hadoop fs -ls /user/hadoop/dir1/filename.txt
            hadoop fs -ls hdfs://<hostname>:9000/user/hadoop/dir1/

Let’s use the following commands as follows and execute. You can ssh to the sandbox using Tools like Putty. You could download putty.exe from the internet.

enter image description here

Let’s touch a file locally.

$ touch filename.txt

enter image description here

Step 2: Now, let’s check how to find out space utilization in a HDFS dir.

hadoop fs -du:
  • Displays sizes of files and directories contained in the given directory or the size of a file if its just a file.
            hadoop fs -du URI
            hadoop fs -du  /user/hadoop/ /user/hadoop/dir1/Sample.txt

enter image description here

Step 4:

Now let’s see how to upload and download files from and to Hadoop Data File System(HDFS)
Upload: ( we have already tried this earlier)

hadoop fs -put:
  • Copy single src file, or multiple src files from local file system to the Hadoop data file system
            hadoop fs -put <localsrc> ... <HDFS_dest_Path>
            hadoop fs -put /home/ec2-user/Samplefile.txt ./ambari.repo /user/hadoop/dir3/

hadoop fs -get:

  • Copies/Downloads files to the local file system
            hadoop fs -get <hdfs_src> <localdst> 
            hadoop fs -get /user/hadoop/dir3/Samplefile.txt /home/

enter image description here

Step 5: Let’s look at quickly two advanced features.

hadoop fs -getmerge
  • Takes a source directory files as input and concatenates files in src into the destination local file.
            hadoop fs -getmerge <src> <localdst> [addnl]
            hadoop fs -getmerge /user/hadoop/dir1/  ./Samplefile2.txt
            addnl: can be set to enable adding a newline on end of each file
hadoop distcp:
  • Copy file or directories recursively
  • It is a tool used for large inter/intra-cluster copying
  • It uses MapReduce to effect its distribution copy, error handling and recovery, and reporting
            hadoop distcp <srcurl> <desturl>
            hadoop distcp hdfs://<NameNode1>:8020/user/hadoop/dir1/ \ 

You could use the following steps to perform getmerge and discp.
Let’s upload two files for this exercise first:

# touch txt1 txt2
# hadoop fs -put txt1 txt2 /user/hadoop/dir2/
# hadoop fs -ls /user/hadoop/dir2/

enter image description here

Step 6:Getting help

You can use Help command to get list of commands supported by Hadoop Data File System(HDFS)

            hadoop fs -help

enter image description here

Hope this short tutorial was useful to get the basics of file management.


Goun Na
May 3, 2014 at 6:09 pm

Very basic and essential commands. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

If you have specific technical questions, please post them in the Forums

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>