Home Forums HDFS Setting up multiple hadoop clusters

This topic contains 4 replies, has 2 voices, and was last updated by  Robert Molina 8 months, 4 weeks ago.

  • Creator
    Topic
  • #47014

    Rajiv Gupta
    Participant

    Hi,
    In our organization, we have hadoop cluster with 13 data nodes and replication factor-2. But we have been facing issue wrt HDD for last two months. There are cases where 3-4 disks are faulty together, leading to data loss.

    Now, we want to set-up separate geographical cluster. Need to know how to synchronize data between these two clusters. Connectivity would not be an issue. But incremental data need to be copied on other cluster.

    Pls suggest.

    Regards,
    Rajiv Gupta

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #49263

    Robert Molina
    Moderator

    Hi Rajiv,
    Sorry about the late response. Distcp would be a feature to use, there are integration tools such as talend. As far as your specific use case where the clusters are distant from each other physically, theoretically it should work. The hortonworks services team may have done something similar in the past. Please feel free to reach out to them if you are still need assistance in designing this use case.

    http://hortonworks.com/contact/

    Regards,
    Robert

    Collapse
    #47242

    Rajiv Gupta
    Participant

    Hi Robert

    Thanks a lot for replying. Yes, we have seen that option, just wanted to see if there is any other better alternative or solution apart from distcp. Is there any limitation for this command to work?

    Regards
    Rajiv

    Collapse
    #47195

    Robert Molina
    Moderator

    Hi Rajiv,
    Have you looked into the Distributed Copy aka distcp feature within hdfs? You can use the command to copy data from one cluster to another.

    Hope that helps.

    Regards,
    Robert

    Collapse
    #47166

    Rajiv Gupta
    Participant

    pls update

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)