Passing parameters to a Map-Reduce program

Passing parameters to a Map-Reduce program

There might be a requirement to pass additional parameters to the mapper and reducers, besides the the inputs which they process.

Setting the parameter:

1. Use the -D command line option to set the parameter while running the job. OR

2. Before launching the job using the old MR API

?

1 2	JobConf job = (JobConf) getConf(); job.set("Amal", "myValue");

3. Before launching the job using the new MR API

?

1 2 3

Configuration conf = new Configuration(); conf.set("Amal", "myValue"); Job job = new Job(conf);

Getting the parameter:

1. Using the old API in the Mapper and Reducer. The JobConfigurable#configure has to be implemented in the Mapper and Reducer class.

?

1 2 3 4

private static Long N; public void configure(JobConf job) {     N = Long.parseLong(job.get("Amal")); }

The variable N can then be used with the map and reduce functions.

2. Using the new API in the Mapper and Reducer. The context is passed to the setup, map, reduce and cleanup functions.

?

1 2	Configuration conf = context.getConfiguration(); String param = conf.get("Amal");