Stdout log capture

to create new topics or reply. | New User Registration

This topic contains 1 reply, has 1 voice, and was last updated by  Dominic Fox 11 months ago.

  • Creator
  • #52352

    Dominic Fox

    I have a WordCount MapReduce job, which tries to write log messages to stdout during Map and Reduce execution, using both log4j’s Logger and System.out.println() statements. Neither output is captured in the job history for the job when I run it against an unaltered HDP 2.1 Sandbox VM instance – in fact, the logs (when viewed through the job history browser) always look like this:

    Log Type: stderr
    Log Length: 222
    log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See for more info.

    Log Type: stdout
    Log Length: 0

    Log Type: syslog
    Log Length: 48335
    […lots of syslog stuff, but none of my log messages]

    Is there some piece of configuration I am missing?

Viewing 1 replies (of 1 total)

You must be to reply to this topic. | Create Account

  • Author
  • #52355

    Dominic Fox

    (here, for reference, is the Wordcount class I’m using)

    package com.opencredo.hadoop.logging;

    import org.apache.log4j.Logger;
    import org.apache.log4j.LogManager;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.mapred.*;

    import java.util.Iterator;
    import java.util.StringTokenizer;

    public class WordCount {

    private static final Logger logger = LogManager.getLogger(WordCount.class);

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
    String line = value.toString();

    logger.warn(“Mapping line ” + line);
    System.out.println(“Mapping line ” + line);
    System.err.println(“Mapping line ” + line);

    StringTokenizer tokenizer = new StringTokenizer(line);
    while (tokenizer.hasMoreTokens()) {
    String nextToken = tokenizer.nextToken();

    logger.warn(“Outputting word ” + nextToken);

    output.collect(word, one);

    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
    logger.error(“Reducing values for ” + key);

    int sum = 0;
    while (values.hasNext()) {
    sum +=;
    logger.error(“Outputting sum ” + sum + ” for key ” + key);
    output.collect(key, new IntWritable(sum));

    public static void main(String[] args) throws Exception {
    JobConf conf = new JobConf(WordCount.class);




    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));


Viewing 1 replies (of 1 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.