merge elements in a Spark RDD under custom condition - mapreduce

Question

How can I merge elements in a Spark RDD under custom condition?
Suppose there is a RDD[Seq[Int]], where some Seq[Int] in this RDD contain overlapping elements. The task is to merge all overlapping Seq[Int] in this RDD, and store the result into a new RDD.
For example, suppose RDD[Seq[Int]] = [[1,2,3], [2,4,5], [1,2], [7,8,9]], the result should be [[1,2,3,4,5], [7,8,9]].
Since RDD[Seq[Int]] is very large, I cannot do it in driver program. Is it possible to get it done using distributed groupBy/map/reduce, etc?

Answer 1

Finally worked it out by myself.
This problem can be transformed into computing all connected components formed by elements in RDD[Seq[Int]], since the merge condition (two Seq[Int] have overlapping integers) denotes connectivity between two Seq[Int].
The basic idea is:
Give each element in RDD[Seq[Int]] an unique key (.zipWithUniqueId)
Group integers in Seq[Int] by the generated key, thus integers that appear in multiple Seq[Int] will have the corresponding key grouped together
Generate a RDD graph, where the edges are key pairs from the same group in Step 2
Use GraphX to compute connected components, and join the results
val sets = Seq(Seq(1,2,3,4), Seq(4,5), Seq(1,2,3), Seq(6,7,8), Seq(9,10), Seq(7,9))
val rddSets = sc.parallelize(sets)
.zipWithUniqueId
.map(x => (x._2, x._1)).cache()
val edges = rddSets.flatMap(s => s._2.map(i => (i, s._1)))
.groupByKey.flatMap(g => {
var first = g._2.head
for (v <- g._2.drop(1)) yield {
val pair = (first, v)
first = v
pair
}
}).flatMap(e => Seq((e._1, e._2), (e._2, e._1)))
val vertices = Graph.fromEdgeTuples[Long](edges, defaultValue = 0)
.connectedComponents.vertices
rddSets.join(vertices).map(x => (x._2._2, x._2._1))
.reduceByKey((s1, s2) => s1.union(s2).distinct)
.collect().foreach(x => println (x._2.toString()))