How to pass an array of integers as a property value in the Configuration object for MapReduce? - mapreduce

Question

We can pass an integer as a Configuration property as below:
Configuration conf = new Configuration();
conf.set("size", 4);
Is there a way to send an array of integers as a property value?
conf.set("list.of.nums", {2, 4, 6, 8}); // one way is to pass them as a String but it doesn't look good

Answer 1

You can serialize the array object into a file and then move the file to HDFS.Then you can add the hdfs file path to Distributed cache using following.
DistributedCache.addCacheFile(new URI(dfsMetaPath + "#"
+ Constants.OBJSYMLINK0), conf);
DistributedCache.createSymlink(conf);
Serialization can be done as follows:-
public static <T> void serializeMetadata(T voObj,
String filePath) throws IOException,NullPointerException {
if(null==voObj){
throw new NullPointerException("NULL object found");
}
ObjectOutputStream oout = null;
FileOutputStream fsout = null;
try {
fsout = new FileOutputStream(filePath);
oout = new ObjectOutputStream(fsout);
oout.writeObject(voObj);
oout.close();
} finally {
if (null != fsout) {
fsout.close();
}
if (null != oout) {
oout.close();
}
}
}
You can use the filepath passed as arguments to the above method.Using this filepath you can move the file to HDFS filepath. Use the HDFS filepath to create the symlink.
To retireve in mapper you can use the following inside setup to get the object back.
File hdfsfilepath = new File(Constants.OBJSYMLINK0);
Integer[] youarray = MetadataSerDeUtils.deserializeMetadata(youarray ,
hdfsfilepath.getAbsolutePath());
For deserialization you can use following code:-
public static <T> T deserializeMetadata(T voObj,
String filePath) throws IOException,NullPointerException, ClassNotFoundException {
FileInputStream fsin = null;
ObjectInputStream oin = null;
try {
fsin = new FileInputStream(filePath);
oin = new ObjectInputStream(fsin);
voObj = (T) oin.readObject();
return voObj;
} finally {
if (null != fsin) {
fsin.close();
}
if (null != oin) {
oin.close();
}
}
}