Home Forums Sqoop Flume multiple collectors

This topic contains 4 replies, has 2 voices, and was last updated by  JuanFra Rodriguez 9 months ago.

  • Creator
    Topic
  • #43981

    JuanFra Rodriguez
    Participant

    Hi:
    I’m wondering what nodes (roles) are more suitable in my hadoop cluster to install Flume collectors.
    I mean, several independent nodes are needed or datanodes could assume this role also?

    Thanks a lot!

    Regards,
    JuanFra.

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Flume multiple collectors’ is closed to new replies.

  • Author
    Replies
  • #44814

    JuanFra Rodriguez
    Participant

    Got it Robert!
    As soon as I have any conclusion, I’ll let you know it.

    Thanks for your support!

    Regards,
    JuanFra.

    Collapse
    #44776

    Robert Molina
    Moderator

    Hi JuanFra,
    It all depends how much data you are fetching and sending to hadoop. If you have a test cluster up, I would suggest running some tests on the cluster and run the flume agent and mimic the amount of data you would be pushing. Watch the resources on the box. Use this data as a baseline and hence you can make a better decision if that particular flume agent can coexist with any other services on that the machine while it is running jobs versus having dedicated machines for flume.

    I hope that helps.

    Regards,
    Robert

    Collapse
    #44093

    JuanFra Rodriguez
    Participant

    Ok, this is our use case:
    We are planning to mount a cluster with the following machines:
    – 1 Management Server
    – 1 Namenode Server
    – 1 Resource Manager
    – 20 Datanodes
    Flume agents will be running in 5 nodes reading system logs.
    Our doubt is about collector tier, where is more suitable to run them? in separated nodes or within datanodes?

    Thanks again!

    Collapse
    #44084

    Robert Molina
    Moderator

    Hi JuanFra,
    What source are you trying to fetch? Can you provide more explanation what you plan to do as far as the use case?

    Regards,
    Robert

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)