pig puzzle: re-writing an involved reducer as a simple pig script? - mapreduce

Question

There are account ids, each with a timestamp grouped by username. foreach of these username groups I want all pairs of (oldest account, other account).
I have a java reducer that does that, can I rewrite it as a simple pig script?
Schema:
{group:(username),A: {(id , create_dt)}
Input:
(batman,{(id1,100), (id2,200), (id3,50)})
(lulu ,{(id7,100), (id9,50)})
Desired output:
(batman,{(id3,id1), (id3,id2)})
(lulu ,{(id9,id7)})

Answer 1

Not that anyone seems to care, but here goes. You have to create a UDF:
desired = foreach my_input generate group as n, FIND_PAIRS(A) as pairs_bag;
And the UDF:
public class FindPairs extends EvalFunc<DataBag> {
#Override
public DataBag exec(Tuple input) throws IOException {
Long pivotCreatedDate = Long.MAX_VALUE;
Long pivot = null;
DataBag accountsBag = (DataBag) input.get(0);
for (Tuple account : accountsBag){
Long accountId = Long.parseLong(account.get(0).toString());
Long creationDate = Long.parseLong(account.get(4).toString());
if (creationDate < pivotCreatedDate ) {
// pivot is the one with the minimal creation_dt
pivot = accountId;
pivotCreatedDate = creationDate;
}
}
DataBag allPairs = BagFactory.getInstance().newDefaultBag();
if (pivot != null){
for (Tuple account : accountsBag){
Long accountId = Long.parseLong(account.get(0).toString());
Long creationDate = Long.parseLong(account.get(4).toString());
if (!accountId.equals(pivot)) {
// we don't want any self-pairs
Tuple output = TupleFactory.getInstance().newTuple(2);
if (pivot < accountId){
output.set(0, pivot.toString());
output.set(1, accountId.toString());
}
else {
output.set(0, accountId.toString());
output.set(1, pivot.toString());
}
allPairs.add(output);
}
}
return allPairs;
}
and if you wanna play real nicely, add this:
/**
* Letting pig know that we emit a bag with tuples, each representing a pair of accounts
*/
#Override
public Schema outputSchema(Schema input) {
try{
Schema pairSchema = new Schema();
pairSchema.add(new FieldSchema(null, DataType.BYTEARRAY));
pairSchema.add(new FieldSchema(null, DataType.BYTEARRAY));
return new Schema(
new FieldSchema(null,
new Schema(pairSchema), DataType.BAG));
}catch (Exception e){
return null;
}
}
}