The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Sqoop Forum

Sqoop Incremental Import

  • #51288
    Swapnil Patil

    I want to import tables from MSSQL Server to HDFS on regular basis (twice a day). How do I do it through sqoop incremental import. Many of my table do not have primary key defined on it. Even tables do not have update date column. Is it possible to import all the tables in one go, coz I have 300 tables that needs to be imported on daily basis.. please help me with this

    Thanks inadvance..

  • Author
  • #51642

    For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job –exec someIncrementalJob will continue to import only newer rows than those previously imported.

    For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work

    Each table must have a single-column primary key.
    You must intend to import all columns of each table.
    You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.


    Patrick Alexander

    I used the following, but every time I run this it loads the entire table and not the incremental

    sqoop import –connect ‘jdbc:sqlserver://;database=mycollection;username=hadoop;password=hadoop’ –table PortalInformation –hive-import — –schema dbo –-incremental -–check-column LineID –last-value 100

    what am I missing ?

    Thanks in advance,


    Hi Patrick,

    Since you are looking to append the table, you would need to use ‘–-incremental append’ with the –check-column.


    Please give this a try and see if you are still seeing the issue.


The topic ‘Sqoop Incremental Import’ is closed to new replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.