Home Forums MapReduce How to count a particular word in a file by taking the word as an argument?

Tagged: 

This topic contains 1 reply, has 2 voices, and was last updated by  abdelrahman 1 year, 2 months ago.

  • Creator
    Topic
  • #32896

    Good Boy
    Member

    Hi Friends,
    I am new to Hadoop mapreduce as well as to java. I am struggling in writing a mapreduce program which will count the number of times a particular word is present in a file. Both the file and the word should be an user input. So I am trying to pass the particular word as an argument to void main() along with the i/p and o/p paths. After getting the word in my void main I need to pass it to my map function to search the occurrence of the word. But I dont know how to do it. Can anyone pls help. Here is my code.

    import java.io.IOException;
    import java.util.Iterator;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
    import org.apache.hadoop.util.GenericOptionsParser;
    public class MyWordCount {
    public static class WordCountMap extends Mapper {
    static String wordToSearch;
    private final static LongWritable ONE = new LongWritable(1L);
    private Text word = new Text();
    public void map(Text key, Text value, Context context)
    throws IOException, InterruptedException {
    if (value.toString().compareTo(wordToSearch) == 0){
    context.write(word, ONE);
    }
    }
    }
    public static class SumReduce extends Reducer {
    public void reduce(Text key, Iterator values,
    Context context) throws IOException, InterruptedException {
    long sum = 0L;
    while (values.hasNext()) {
    sum += values.next().get();
    }
    context.write(key, new LongWritable(sum));
    }
    }
    public static void main(String[] rawArgs) throws Exception {
    GenericOptionsParser parser = new GenericOptionsParser(rawArgs);
    Configuration conf = parser.getConfiguration();
    String[] args = parser.getRemainingArgs();
    Job job = new Job(conf, “wordcount”);
    job.setJarByClass(WordCountMap.class);
    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(LongWritable.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(LongWritable.class);
    job.setMapperClass(WordCountMap.class);
    job.setReducerClass(SumReduce.class);
    job.setInputFormatClass(SequenceFileInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    String myWord = args[2];
    job.waitForCompletion(true);
    }
    }
    I need to get the value of “myWord” from main() function to map() function.
    Thanks in advance
    Biswa

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #33386

    abdelrahman
    Moderator

    Hi,

    Please use it as implemented in the following:

    public int run(String[] args) throws Exception {
    Configuration conf = getConf();
    args = new GenericOptionsParser(conf, args).getRemainingArgs();

    // Get the input name as arguments
    String WordCount = args[0];
    }

    Thanks
    -Abdelrahman

    Collapse
Viewing 1 replies (of 1 total)