Home Forums MapReduce MapReduce issue

Tagged: 

This topic contains 5 replies, has 4 voices, and was last updated by  abdelrahman 1 year ago.

  • Creator
    Topic
  • #29833

    Hi,

    I am new to Hadoop concepts.
    While practicing with one custom MapReduce program, I found the result is not as expected after executing the code on HDFS based file. Please note that when I execute the same program using Unix based file,getting expected result.
    Below are the details of my code.

    MapReduce in java
    ==================

    import java.io.IOException;
    import java.util.*;

    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.conf.*;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapred.*;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.util.*;

    public class WordCount1 {

    public static class Map extends MapReduceBase implements Mapper {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) throws IOException {
    String line = value.toString();
    String tokenedZone=null;
    StringTokenizer tokenizer = new StringTokenizer(line);
    while (tokenizer.hasMoreTokens()) {
    tokenedZone=tokenizer.nextToken();
    word.set(tokenedZone);
    output.collect(word, one);
    }
    }
    }

    public static class Reduce extends MapReduceBase implements Reducer {
    public void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException {
    int sum = 0;
    int val = 0;
    while (values.hasNext()) {
    val = values.next().get();
    sum += val;
    }
    if(sum>1)
    output.collect(key, new IntWritable(sum));
    }
    }

    public static void main(String[] args) throws Exception {
    JobConf conf = new JobConf();
    conf.setJarByClass(WordCount1.class);
    conf.setJobName(“wordcount1″);

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(Map.class);
    conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);

    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);

    Path inPath = new Path(args[0]);
    Path outPath = new Path(args[0]);

    FileInputFormat.setInputPaths(conf,inPath );
    FileOutputFormat.setOutputPath(conf, outPath);

    JobClient.runJob(conf);
    }

    }

    input File
    ===========
    test my program
    during test and my hadoop
    your during
    get program

    hadoop generated output file on HDFS file system
    =======================================
    during 2
    my 2
    test 2

    hadoop generated output file on local file system
    =======================================
    during 2
    my 2
    program 2
    test 2

    Please help me on this issue

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #33399

    abdelrahman
    Moderator

    Hi,

    Please start with the basic M-R word count program. And alter the configuration as you see fit later. Can you post the complete error.

    http://wiki.apache.org/hadoop/WordCount

    Thanks
    -Abdelrahman

    Collapse
    #31670


    Member

    Hello, I have a question, in my program i have one error in JobClient.runJob(conf);
    Ps: I use hadoop-0.19.1
    Thank you

    Collapse
    #29939

    tedr
    Moderator

    Hi Mullangi,

    do the files in hdfs have the same data in them? you can verify this with the command ‘hadoop fs -cat <filename>’ where ‘<filename>’ is the path to the file in hdfs including the file itself.

    Thanks,
    Ted.

    Collapse
    #29933

    Hi Ted,

    This is happening to me even when I changed to another file.

    Thanks,
    Ram

    Collapse
    #29865

    tedr
    Moderator

    HI Mullangi,

    Check that the file in HDFS actually matches the file on the local system and has not been truncated/corrupted in some way.

    Thanks,
    Ted.

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)