Want to extract my server logs to HDFS

Is it Flume is the only solution? I was able to install Flume on Sandbox but I got sutck at configuration. Any help would be appreciated.

Thanks.

Topic

[Larry LiuModerator]

Hi, Mur,

I think the first thing to do is to set up flume agent on SQL servers.

I am referring to the following web site:

http://flume.apache.org/FlumeUserGuide.html#configuring-a-multi-agent-flow

Please let us know when you have questions during configuration.

Larry

[Mur RaguthuMember]

Hi Larry,
As of now we are able to pull messages from sandbox file system to HDFS (same server) via Flume-NG. We have multiple SQL servers and would like to pull those server logs to HDFS for analysis. That is the plan..

Is that makes sense? Please let us know.

[Larry LiuModerator]

Hi, Mur,

Can you please clarify what you are trying to do? From your last post, you are planning to extract other server logs in to Sandbox HDFS. I am wondering if you could provide more detail? We can start from here.

Thanks
Larry

[Mur RaguthuMember]

We are able to set up Flume-NG in Sandbox and able to load ‘sandbox log files’ from same sandbox to HDFS. Now we are planning to extract other server logs in to Sandbox HDFS. Yes I am going through documentation and planning to set up like “$ telnet localhost 44444”. Did anybody done this with Sandbox? Share your experiences please?

[tedrModerator]

Hi Mur,

Thanks for trying the Hortonworks Sandbox.
Am I correct in assuming that you’ve read over the documentation at : http://flume.apache.org/FlumeUserGuide.html ?

Thanks,
Ted.

[Author]

Replies