using mapreduce to find common items between users - mapreduce

Question

Suppose I have the following user/item sets where items could also be replicates for each user (like user1)
{ "u1", "item" : [ "a", "a", "c","h" ] }
{ "u2", "item" : [ "b", "a", "f" ] }
{ "u3", "item" : [ "a", "a", "f" ] }
I want to find a map-reduce algorithm that will calculate the number of common items between each pair of users some like that
{ "u1_u2", "common_items" : 1 }
{ "u1_u3", "common_items" : 2 }
{ "u2_u3", "common_items" : 2 }
It basically finds the intersections of itemsets for each pair and considers replicates as new items. I am new to mapreduce, how can I do a map-reduce for this?

Answer 1

With these sorts of problems, you need to appreciate that some algorithms will scale better than others, and performance of any one algorithm will depend on the 'shape' and size of your data.
Comparing the item sets for every user to every other user might be appropriate for small domain datasets (say 1000's or users, maybe even 10,000's, with a similar number of items), but is an 'n squared' problem (or an order of thereabouts, my Big O is rusty to say the least!):
Users Comparisons
----- -----------
2 1
3 3
4 6
5 10
6 15
n (n^2 - n)/2
So a user domain of 100,000 would yield 4,999,950,000 set comparisons.
Another approach to this problem, would be to inverse the relationship, so run a Map Reduce job to generate a map of items to users:
'a' : [ 'u1', 'u2', 'u3' ],
'b' : [ 'u2' ],
'c' : [ 'u1' ],
'f' : [ 'u2', 'u3' ],
'h' : [ 'u1' ],
From there you can iterate the users for each item and output user pairs (with a count of one):
'a' would produce: [ 'u1_u2' : 1, 'u1_u3' : 1, 'u2_u3' : 1 ]
'f' would produce: [ 'u2_u3' : 1 ]
Then finally produce the sum for each user pairing:
[ 'u1_u2' : 1, 'u1_u3' : 1, 'u2_u3' : 2 ]
This doesn't produce the behavior you are interested (the double a's in both u1 and u3 item sets), but details an initial implementation.
If you know your domain set typically has users which do not have items in common, a small number of items per user, or an item domain which has a large number of distinct values, then this algorithm will be more efficient (previously you were comparing every user to another, with a low probability of intersection between the two sets). I'm sure a mathematician could prove this for you, but that i am not!
This also has the same potential scaling problem as before - in that if you have an item that all 100,000 users all have in common, you still need to generate the 4 billion user pairs. This is why it is important to understand your data, before blindly applying an algorithm to it.

Answer 2

You want a step that emits all of the things the user has, like:
{ 'a': "u1" }
{ 'a': "u1" }
{ 'c': "u1" }
{ 'h': "u1" }
{ 'b': "u2" }
{ 'a': "u2" }
{ 'f': "u2" }
{ 'a': "u1" }
{ 'a': "u3" }
{ 'f': "u3" }
Then reduce them by key like:
{ 'a': ["u1", "u1", "u2", "u3"] }
{ 'b': ["u2"] }
{ 'c': ["u1"] }
{ 'f': ["u2", "u3"] }
{ 'h': ["u1"] }
And in that reducer emit the permutations of each user in each value, like:
{ 'u1_u2': 'a' }
{ 'u2_u3': 'a' }
{ 'u1_u3': 'a' }
{ 'u2_u3': 'f' }
Note that you'll want to make sure that in a key like k1_k2 that k1 < k2 so that they match up in any further mapreduce steps.
Then if if you need them all grouped like your example, another mapreduce phase to combine them by key and they'll end up like:
{ 'u1_u2': ['a'] }
{ 'u1_u3': ['a'] }
{ 'u2_u3': ['a', 'f'] }
{ 'u2_u3': ['f'] }

Answer 3

Does this work for you?
from itertools import combinations
user_sets = [
{ 'u1': [ 'a', 'a', 'c', 'h' ] },
{ 'u2': [ 'b', 'a', 'f' ] },
{ 'u3': [ 'a', 'a', 'f' ] },
]
def compare_sets(set1, set2):
sum = 0
for n, item in enumerate(set1):
if item in set2:
sum += 1
del set2[set2.index(item)]
return sum
for set in combinations(user_sets, 2):
comp1, comp2 = set[0], set[1]
print 'Common items bwteen %s and %s: %s' % (
comp1.keys()[0], comp2.keys()[0],
compare_sets(comp1.values()[0], comp2.values()[0])
)
Here's the output:
Common items bwteen u1 and u2: 1
Common items bwteen u1 and u3: 2
Common items bwteen u2 and u3: 1