Windows 2012 STD R2 Server – Flume Errors

to create new topics or reply. | New User Registration

This topic contains 1 reply, has 1 voice, and was last updated by  S R 1 year, 3 months ago.

  • Creator
  • #51386

    S R

    I am getting the following error message and it goes on a loop. Appreciate if someone has already fixed. I am using agent Source as spoolDir. The Syslog is sitting in the Ingest folder and also I see data.<Number>.seq file in HDFS

    09 Apr 2014 14:57:51,469 ERROR [pool-9-thread-1] (org.apache.flume.source.SpoolDirectorySource$ – Uncaught exception in Runnable
    java.lang.IllegalStateException: Serializer has been closed
    at org.apache.flume.serialization.LineDeserializer.ensureOpen(
    at org.apache.flume.serialization.LineDeserializer.readEvents(
    at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(
    at org.apache.flume.source.SpoolDirectorySource$
    at java.util.concurrent.Executors$
    at java.util.concurrent.FutureTask.runAndReset(
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
    at java.util.concurrent.ScheduledThreadPoolExecutor$
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    at java.util.concurrent.ThreadPoolExecutor$

    Software Versions
    HDP Version : 2.0
    OS : Windows 2012 Standard Server R2

    Flume Configuration File:
    # Name the components on this agent
    agent.sources = WinHadoopC1Source
    agent.sinks = WinHadoopC1Sink1
    agent.channels = WinHadoopC1Channel1

    # Describe/configure the source
    agent.sources.WinHadoopC1Source.spoolDir = C:/flume_spooldir
    agent.sources.WinHadoopC1Source.fileHeader = true

    # Describe the sink
    agent.sinks.WinHadoopC1Sink1.hdfs.path = hdfs://WIN-ATKSGSRL5DL/logspooldir
    agent.sinks.WinHadoopC1Sink1.hdfs.fileType = SequenceFile
    agent.sinks.WinHadoopC1Sink1.hdfs.filePrefix = data
    agent.sinks.WinHadoopC1Sink1.hdfs.fileSuffix = .seq

    # Use a channel which buffers events in memory
    agent.channels.WinHadoopC1Channel1.type = memory
    agent.channels.WinHadoopC1Channel1.capacity = 100000
    agent.channels.WinHadoopC1Channel1.transactionCapacity = 10000

    # Bind the source and sink to the channel
    agent.sources.WinHadoopC1Source.channels = WinHadoopC1Channel1 = WinHadoopC1Channel1
    Thanks, Satya Raju

Viewing 1 replies (of 1 total)

You must be to reply to this topic. | Create Account

  • Author
  • #51844

    S R

    Was able to test successfully on Windows single node installation. Gave Read/Write permissions on the Spool Dir for Hadoop User and after that it started working. Interestingly I went back and removed the ‘Hadoop’ user Object from that Folder and still it works. It works but not sure of the root cause.

Viewing 1 replies (of 1 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.