Home Forums Sqoop Sqoop Incremental Import

This topic contains 3 replies, has 3 voices, and was last updated by  Mahesh Balakrishnan 6 months ago.

  • Creator
    Topic
  • #51288

    Swapnil Patil
    Participant

    Hi…
    I want to import tables from MSSQL Server to HDFS on regular basis (twice a day). How do I do it through sqoop incremental import. Many of my table do not have primary key defined on it. Even tables do not have update date column. Is it possible to import all the tables in one go, coz I have 300 tables that needs to be imported on daily basis.. please help me with this

    Thanks inadvance..

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Sqoop Incremental Import’ is closed to new replies.

  • Author
    Replies
  • #54699

    Hi Patrick,

    Since you are looking to append the table, you would need to use ‘–-incremental append’ with the –check-column.

    Refer: http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports.

    Please give this a try and see if you are still seeing the issue.

    -Mahesh

    Collapse
    #53198

    Patrick Alexander
    Participant

    I used the following, but every time I run this it loads the entire table and not the incremental

    sqoop import –connect ‘jdbc:sqlserver://10.10.100.62:4101;database=mycollection;username=hadoop;password=hadoop’ –table PortalInformation –hive-import — –schema dbo –-incremental -–check-column LineID –last-value 100

    what am I missing ?

    Thanks in advance,
    Patrick

    Collapse
    #51642

    For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job –exec someIncrementalJob will continue to import only newer rows than those previously imported.

    For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work

    Each table must have a single-column primary key.
    You must intend to import all columns of each table.
    You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.

    -Mahesh

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)