Using the command line to manage files on HDFS

In this tutorial we will walk through some of the basic HDFS commands you will need to manage files on HDFS. To complete this tutorial you will need a working HDP cluster. The easiest way to have a Hadoop cluster is to download the Hortonworks Sandbox.

Let’s get started.

Step 1: Let’s create a directory in HDFS, upload a file and list.

Let’s look at the syntax first:

hadoop fs -mkdir:
  • It will take path uri’s as argument and creates directory or directories.
    Usage: 
            hadoop fs -mkdir <paths> 
    Example:
            hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
            hadoop fs -mkdir hdfs://nn1.example.com/user/hadoop/dir
hadoop fs -ls:
  • Lists the contents of a directory
  • For a file returns stats of a file
    Usage: 
            hadoop fs -ls <args>
    Example: 
            hadoop fs -ls /user/hadoop/dir1 /user/hadoop/dir2
            hadoop fs -ls /user/hadoop/dir1/filename.txt
            hadoop fs -ls hdfs://<hostname>:9000/user/hadoop/dir1/

Let’s use the following commands as follows and execute. You can ssh to the sandbox using Tools like Putty. You could download putty.exe from the internet.

enter image description here

Let’s touch a file locally.

$ touch filename.txt

enter image description here

Step 2: Now, let’s check how to find out space utilization in a HDFS dir.

hadoop fs -du:
  • Displays sizes of files and directories contained in the given directory or the size of a file if its just a file.
    Usage:  
            hadoop fs -du URI
    Example:
            hadoop fs -du  /user/hadoop/ /user/hadoop/dir1/Sample.txt

enter image description here

Step 4:

Now let’s see how to upload and download files from and to Hadoop Data File System(HDFS)
Upload: ( we have already tried this earlier)

hadoop fs -put:
  • Copy single src file, or multiple src files from local file system to the Hadoop data file system
    Usage: 
            hadoop fs -put <localsrc> ... <HDFS_dest_Path>
    Example:
            hadoop fs -put /home/ec2-user/Samplefile.txt ./ambari.repo /user/hadoop/dir3/

Download:
hadoop fs -get:

  • Copies/Downloads files to the local file system
    Usage: 
            hadoop fs -get <hdfs_src> <localdst> 
    Example:
            hadoop fs -get /user/hadoop/dir3/Samplefile.txt /home/

enter image description here

Step 5: Let’s look at quickly two advanced features.

hadoop fs -getmerge
  • Takes a source directory files as input and concatenates files in src into the destination local file.
    Usage: 
            hadoop fs -getmerge <src> <localdst> [addnl]
    Example:
            hadoop fs -getmerge /user/hadoop/dir1/  ./Samplefile2.txt
    Option:
            addnl: can be set to enable adding a newline on end of each file
hadoop distcp:
  • Copy file or directories recursively
  • It is a tool used for large inter/intra-cluster copying
  • It uses MapReduce to effect its distribution copy, error handling and recovery, and reporting
    Usage: 
            hadoop distcp <srcurl> <desturl>
    Example:
            hadoop distcp hdfs://<NameNode1>:8020/user/hadoop/dir1/ \ 
                    hdfs://<NameNode2>:8020/user/hadoop/dir2/

You could use the following steps to perform getmerge and discp.
Let’s upload two files for this exercise first:

# touch txt1 txt2
# hadoop fs -put txt1 txt2 /user/hadoop/dir2/
# hadoop fs -ls /user/hadoop/dir2/

enter image description here

Step 6:Getting help

You can use Help command to get list of commands supported by Hadoop Data File System(HDFS)

    Example:    
            hadoop fs -help

enter image description here

Hope this short tutorial was useful to get the basics of file management.

Comments

Amey
|
September 24, 2014 at 5:52 am
|

How to mount HDFS in local system? I am using HDP 2.1 sandbox on virtualbox ready made image downloaded from this site.
I tried multiple ways but getting errors.

Adam
|
September 23, 2014 at 11:21 am
|

Thanks. I am starting to learn about Hadoop after almost 20 years of Oracle/DBA experience. What is the best starting book for administrators? not developers – I know you have to have some developer background but I really mainly focus on architecture and design.
Thanks again this was good start for me.

    Jules S. Damji
    |
    September 24, 2014 at 2:38 pm
    |

    There are number of books for DevOps and Administrators. A good book to begin with is Hadoop Operations. Since you come from a Relation DB background and most of data repositories in Hadoop are NoSQL, you might want read up on NoSQL concepts and semantics, for the access and storage patterns are different.

    Wiki pages for NoSQL databases like MongoDB, Casandra, HBase, etc are a good start. Another way to get familiar with Hadoop, is download Hortonworks Sandbox. We have some tutorials targeted toward Administrators.

    Cheers
    Jules

Goun Na
|
May 3, 2014 at 6:09 pm
|

Very basic and essential commands. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

If you have specific technical questions, please post them in the Forums

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>