Falcon Forum

disctp cant talk to target cluster

  • #58192
    Cameron Walker
    Participant

    We’re looking at backup methods for our hadoop. We found out quickly that disctp will not work to copy data between our clusters because the Name nodes have been configured on networks that are private to each cluster. Therefore the datanodes from our primary cluster can not talk to the namenodes on the target cluster.

    However, our edge nodes for our two clusters have been configured so that they can, in fact, talk to both private networks. The network designers have placed nice 1G pipes on the edgenode hosts for the purpose of allowing the edge nodes to move data in and out of the clusters.

    The network team is of the opinion that Falcon can extract data from the datanodes in cluster through the single edgenode and transfer the data to the target cluster. My understanding is that Falcon is built on top of distcp and, if distcp can’t move the data, then Falcon can’t move the data.

    Who is correct? Can Falcon funnel all the data in a transfer through a single edge node like they believe? [ I realize that, even if possible, it presents a bottleneck but that’s a different issue right now; the question is can it be done?]

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #64129
    Participant

    Vous pouvez facilement trouver vrais Moncler pas chère à prix réduits. Moncler? Doudoune Femme jolie et colorée,http://www.rhumatologie-grenoble.com/moncler.asp – Doudoune Moncler Pas Che boutique Moncler d’avoir une réputation de longue date. 2013 Veste Doudoune Moncler Femmes & Homme Pas Cher en ligne vente.,Acheter Amende UGG Pas Cher à Bas Prix Et Un Service Remarquable à La Bottes UGG Enhttp://www.corevihouest.org/uggs.asp – bottes pas cher Ligne Avec La Livraison Rapide Dans Le Monde Entie.,Acheter Amende UGG Pas Cher à Bas Prix Et Un Service Remarquable à La Bottes UGG Enhttp://www.corevihouest.org/uggs.asp – ugg pas chere Ligne Avec La Livraison Rapide Dans Le Monde Entie.,Vous pouvez facilement trouver vrais Moncler pas chère à prix réduits. Moncler? Doudoune Femme jolie et colorée,http://www.rhumatologie-grenoble.com/moncler.asp – Doudoune Moncler boutique Moncler d’avoir une réputation de longue date. 2013 Veste Doudoune Moncler Femmes & Homme Pas Cher en ligne vente.

    #71475
    Rama Balam
    Moderator

    Can someone look into this question? I also have a similar situation and would like to know the answer/approach to solve this problem.

    #72291
    Sanjeev T
    Participant

    falcon uses distcp, for replication.

    you need to open the connectivity between two clusters for feed replication.

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.