Home Forums Sqoop Flume integration with Storm

This topic contains 2 replies, has 2 voices, and was last updated by  Robert Molina 9 months, 2 weeks ago.

  • Creator
    Topic
  • #42946

    Flume sink does the push mechanism, while Storm spout does the pull mechanism for the events. Any pointers for getting Flume working with Storm, so we can use one event collection tool?
    The other tool Kafka provides the right integration for Storm, but HDP bundles the Flume by default.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Flume integration with Storm’ is closed to new replies.

  • Author
    Replies
  • #49597

    Robert Molina
    Moderator

    Hi Satish,
    From what I gathered, storm is stateless and an external system is needed to have the messages persisted while the data gets processed. This persistent piece is Kafka, where external sources push their data to. On the other hand, flume is designed to stream data, where it receives data on the source and pushes it to the target endpoint via its attached sink. This is different from Kafka where another system will pull from it. Based on that, flume may not be a good substitute for kafka within the storm and Kafka use case. Although, there seems to be some information on the web about a kafka sink for flume, in which Flume can collect data from multiple sources and deposit it into a Kafka cluster that will eventually be picked up by Storm. As of this time, kafka sink is not yet part of the HDP distro or Apache for flume.
    Hope the information helps.

    Regards,
    Robert

    Collapse
    #46477

    Robert Molina
    Moderator

    Hi Satish,
    I haven’t heard any uses case with Flume and Storm, will do some research and see what I can find. What is the use case you are trying to implement?

    Kind Regards,
    Robert

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)