Column recoding based on count of distincts - python-2.7

Question

I've a panda data frame like this:
import pandas as pd
data = {'VAR1': ['A', 'A', 'A', 'A','B', 'B'],
'VAR2': ['C', 'V', 'C', 'C','V', 'D']}
frame = pd.DataFrame(data)
Fundamentally I need to recode each variable. The recoding would work like this: calculate a count of distinct values for each column, and if the count is greater than or equal to a threshold, keep the original value, otherwise set a new value of 'X'. If the threshold were 3, then this is what it would need to look like.
data2 = {'VAR3': ['A', 'A', 'A', 'A','X', 'X'],
'VAR4': ['C', 'X', 'C', 'C','X', 'X']}
frame2 = pd.DataFrame(data2)
And this is the desired output, with the original data merged to the recoded data.
pd.merge(frame, frame2, left_index=True, right_index=True)
I'm new to Python and while the book Python for Data Analysis is really helping me, I still cannot quite figure out how to achieve the desired result in a simple way.
Any help would be appreciated!

Answer 1

Take each column individually. Group it by value, and use the filter method on groups to replace any group with less than 3 values with NaN. Then replace those NaNs with X.
You could do this all in one list comprehension, but for clarity I defined a recode function that does all the substantial stuff.
In [38]: def recode(s, threshold):
....: return s.groupby(s).filter(lambda x: x.count() >= threshold, dropna=False).fillna(value='X')
....:
Applying to each column and then reassembling the columns into one new DataFrame....
In [39]: frame2 = pd.concat([recode(frame[col], 3) for col in frame], axis=1)
In [40]: frame2
Out[40]:
VAR1 VAR2
0 A C
1 A X
2 A C
3 A C
4 X X
5 X X
And, to be sure, you can merge the original and the recoded frames just as you expressed it in your question:
In [27]: pd.merge(frame, frame2, left_index=True, right_index=True)
Out[27]:
VAR1_x VAR2_x VAR1_y VAR2_y
0 A C A C
1 A V A X
2 A C A C
3 A C A C
4 B V X X
5 B D X X
Edit: Use this equivalent workaround for pandas version < 0.12:
def recode(s, threshold):
b = s.groupby(s).transform(lambda x: x.count() >= threshold).astype('bool') # True/False
s[~b] = 'X'
return s