From a given array (call it numbers[]), i want another array (results[]) which contains all sum possibilities between elements of the first array.
For example, if I have numbers[] = {1,3,5}, results[] will be {1,3,5,4,8,6,9,0}.
there are 2^n possibilities.
It doesn't matter if a number appears two times because results[] will be a set
I did it for sum of pairs or triplet, and it's very easy. But I don't understand how it works when we sum 0, 1, 2 or n numbers.
This is what I did for pairs :
std::unordered_set<int> pairPossibilities(std::vector<int> &numbers) {
std::unordered_set<int> results;
for(int i=0;i<numbers.size()-1;i++) {
for(int j=i+1;j<numbers.size();j++) {
results.insert(numbers.at(i)+numbers.at(j));
}
}
return results;
}
Also, assuming that the numbers[] is sorted, is there any possibility to sort results[] while we fill it ?
Thanks!
This can be done with Dynamic Programming (DP) in O(n*W) where W = sum{numbers}.
This is basically the same solution of Subset Sum Problem, exploiting the fact that the problem has optimal substructure.
DP[i, 0] = true
DP[-1, w] = false w != 0
DP[i, w] = DP[i-1, w] OR DP[i-1, w - numbers[i]]
Start by following the above solution to find DP[n, sum{numbers}].
As a result, you will get:
DP[n , w] = true if and only if w can be constructed from numbers
Following on from the Dynamic Programming answer, You could go with a recursive solution, and then use memoization to cache the results, top-down approach in contrast to Amit's bottom-up.
vector<int> subsetSum(vector<int>& nums)
{
vector<int> ans;
generateSubsetSum(ans,0,nums,0);
return ans;
}
void generateSubsetSum(vector<int>& ans, int sum, vector<int>& nums, int i)
{
if(i == nums.size() )
{
ans.push_back(sum);
return;
}
generateSubsetSum(ans,sum + nums[i],nums,i + 1);
generateSubsetSum(ans,sum,nums,i + 1);
}
Result is : {9 4 6 1 8 3 5 0} for the set {1,3,5}
This simply picks the first number at the first index i adds it to the sum and recurses. Once it returns, the second branch follows, sum, without the nums[i] added. To memoize this you would have a cache to store sum at i.
I would do something like this (seems easier) [I wanted to put this in comment but can't write the shifting and removing an elem at a time - you might need a linked list]
1 3 5
3 5
-----
4 8
1 3 5
5
-----
6
1 3 5
3 5
5
------
9
Add 0 to the list in the end.
Another way to solve this is create a subset arrays of vector of elements then sum up each array's vector's data.
e.g
1 3 5 = {1, 3} + {1,5} + {3,5} + {1,3,5} after removing sets of single element.
Keep in mind that it is always easier said than done. A single tiny mistake along the implemented algorithm would take a lot of time in debug to find it out. =]]
There has to be a binary chop version, as well. This one is a bit heavy-handed and relies on that set of answers you mention to filter repeated results:
Split the list into 2,
and generate the list of sums for each half
by recursion:
the minimum state is either
2 entries, with 1 result,
or 3 entries with 3 results
alternatively, take it down to 1 entry with 0 results, if you insist
Then combine the 2 halves:
All the returned entries from both halves are legitimate results
There are 4 additional result sets to add to the output result by combining:
The first half inputs vs the second half inputs
The first half outputs vs the second half inputs
The first half inputs vs the second half outputs
The first half outputs vs the second half outputs
Note that the outputs of the two halves may have some elements in common, but they should be treated separately for these combines.
The inputs can be scrubbed from the returned outputs of each recursion if the inputs are legitimate final results. If they are they can either be added back in at the top-level stage or returned by the bottom level stage and not considered again in the combining.
You could use a bitfield instead of a set to filter out the duplicates. There are reasonably efficient ways of stepping through a bitfield to find all the set bits. The max size of the bitfield is the sum of all the inputs.
There is no intelligence here, but lots of opportunity for parallel processing within the recursion and combine steps.
So I need an algorithm to generate all permutations of a list of numbers excluding cyclic rotations (e.g. [1,2,3] == [2,3,1] == [3,1,2]).
When there is at least 1 unique number in the sequence it is fairly straight forward, take out that unique number, generate all permutations of the remaining numbers (but with a small modification to the 'standard' permutations algorithm) and add the unique number to the front.
For generating the permutations I've found that it's necessary to change the permutations code to:
def permutations(done, options)
permuts = []
seen = []
for each o in options
if o not in seen
seen.add(o)
permuts += permutations(done+o, options.remove(o))
return permuts
Only using each unique number in options once means that you don't get 322 twice.
This algorithm still outputs rotations when there are no unique elements, e.g. for [1,1,2,2] it would output [1,1,2,2], [1,2,2,1] and [1,2,1,2] and the first two are cyclic rotations.
So is there an efficient algorithm that would allow me to generate all the permutations without having to go through afterwards to remove cyclic rotations?
If not, what would be the most efficient way to remove cyclic rotations?
NOTE: this is not using Python, but rather C++.
For the case of permutations where all numbers are distinct, this is simple. Say the numbers are 1,2,...,n, then generate all permutations of 1,2,...,n-1 and stick n at the front. This gives all permutations of the full set modulo cyclic rotations. For example, with n=4, you would do
4 1 2 3
4 1 3 2
4 2 1 3
4 2 3 1
4 3 1 2
4 3 2 1
This ensures that some cyclic rotation of each permutation of 1,2,3,4 appears exactly once in the list.
For the general case where you want permutations of a multiset (repeated entries allowed), you can use a similar trick. Remove all instances of the largest letter n (similar to ignoring the 4 in the above example) and generate all permutations of the remaining multiset. The next step is to put the ns back into the permutations in canonical ways (similar to putting the 4 at the beginning in the above example).
This is really just a case of finding all Lyndon words to generate necklaces
Think about testing each of the permutations you output, looking for a cyclic rotation that's "lexically" earlier than the one you've got. If there is one, don't return it - it will have been enumerated somewhere else.
Choosing a "unique" first element, if one exists, helps you optimize. You know if you fix the first element, and it's unique, then you can't possibly have duplicated it with a rotation. On the other hand, if there's no such unique element, just choose the one that occurs the least. That way you only need to check for cyclic rotations that have that first element. (Example, when you generate [1,2,2,1] - you only need to check [1,1,2,2], not [2,2,1,1] or [2,1,1,2]).
OK, pseudocode... clearly O(n!), and I'm convinced there's no smarter way, since the case "all symbols different" obviously has to return (n-1)! elements.
// generate all permutations with count[0] 0's, count[1] 1's...
def permutations(count[])
if(count[] all zero)
return just the empty permutation;
else
perms = [];
for all i with count[i] not zero
r = permutations(copy of count[] with element i decreased);
perms += i prefixed on every element of r
return perms;
// generate all noncyclic permutations with count[0] 0's, count[1] 1's...
def noncyclic(count[])
choose f to be the index with smallest count[f];
perms = permutations(copy of count[] with element f decreased);
if (count[f] is 1)
return perms;
else
noncyclic = [];
for each perm in perms
val = perm as a value in base(count.length);
for each occurence of f in perm
test = perm rotated so that f is first
tval = test as a value in base(count.length);
if (tval < val) continue to next perm;
if not skipped add perm to noncyclic;
return noncyclic;
// return all noncyclic perms of the given symbols
def main(symbols[])
dictionary = array of all distinct items in symbols;
count = array of counts, count[i] = count of dictionary[i] in symbols
nc = noncyclic(count);
return (elements of nc translated back to symbols with the dictionary)
This solution is going to involve a bit of itertools.permutations usage, set(), and some good ol' fashioned set difference. Bear in mind, the runtime for finding a permutation will still be O(n!). My solution won't do it in-line, either, but there may be a much more elegant solution that allows you to do so (and doesn't involve itertools.permutations). For this purpose, this is the straightforward way to accomplish the task.
Step 1: Algorithm for generating cycles, using the first element given. For a list [1, 1, 2, 2], this will give us [1, 1, 2, 2], [1, 2, 2, 1], [2, 1, 1, 2], [2, 2, 1, 1].
def rotations(li):
count = 0
while count < len(li):
yield tuple(li)
li = li[1:] + [li[0]]
count += 1
Step 2: Importing itertools.permutations to give us the permutations in the first place, then setting up its results into a set.
from itertools import permutations
perm = set(permutations([1, 1, 2, 2]))
Step 3: Using the generator to give us our own set, with the cycles (something we want to rid ourselves of).
cycles = set(((i for i in rotations([1, 1, 2, 2]))))
Step 4: Apply set difference to each and the cycles are removed.
perm = perm.difference(cycles)
Hopefully this will help you out. I'm open to suggestions and/or corrections.
First I'll show the containers and algorithms we'll be using:
#include <vector>
#include <set>
#include <algorithm>
#include <iostream>
#include <iterator>
using std::vector;
using std::set;
using std::sort;
using std::next_permutation;
using std::copy;
using std::ostream_iterator;
using std::cout;
Next our vector which will represent the Permutation:
typedef vector<unsigned int> Permutation;
We need a comparison object to check whether a permutation is a rotation:
struct CompareCyclicPermutationsEqual
{
bool operator()(const Permutation& lhs, const Permutation& rhs);
};
And typedef a set which uses the cyclic comparison object:
typedef set<Permutation, CompareCyclicPermutationsEqual> Permutations;
Then the main function is quite simple:
int main()
{
Permutation permutation = {1, 2, 1, 2};
sort(permutation.begin(). permutation.end());
Permutations permutations;
do {
permutations.insert(permutation);
} while(next_permutation(numbers.begin(), numbers.end()))
copy(permutations.begin(), permutations.end(),
ostream_iterator<Permutation>(cout, "\n");
return 0;
}
Output:
1, 1, 2, 2,
1, 2, 1, 2,
I haven't implemented CompareCyclicPermutationsEqual yet. Also you'd need to implement ostream& operator<<(ostream& os, const Permutation& permutation).
I have an application where I have a number of sets. A set might be
{4, 7, 12, 18}
unique numbers and all less than 50.
I then have several data items:
1 {1, 2, 4, 7, 8, 12, 18, 23, 29}
2 {3, 4, 6, 7, 15, 23, 34, 38}
3 {4, 7, 12, 18}
4 {1, 4, 7, 12, 13, 14, 15, 16, 17, 18}
5 {2, 4, 6, 7, 13, 15}
Data items 1, 3 and 4 match the set because they contain all items in the set.
I need to design a data structure that is super fast at identifying whether a data item is a member of a set includes all the members that are part of the set (so the data item is a superset of the set). My best estimates at the moment suggest that there will be fewer than 50,000 sets.
My current implementation has my sets and data as unsigned 64 bit integers and the sets stored in a list. Then to check a data item I iterate through the list doing a ((set & data) == set) comparison. It works and it's space efficient but it's slow (O(n)) and I'd be happy to trade some memory for some performance. Does anyone have any better ideas about how to organize this?
Edit:
Thanks very much for all the answers. It looks like I need to provide some more information about the problem. I get the sets first and I then get the data items one by one. I need to check whether the data item is matches one of the sets.
The sets are very likely to be 'clumpy' for example for a given problem 1, 3 and 9 might be contained in 95% of sets; I can predict this to some degree in advance (but not well).
For those suggesting memoization: this is this the data structure for a memoized function. The sets represent general solutions that have already been computed and the data items are new inputs to the function. By matching a data item to a general solution I can avoid a whole lot of processing.
I see another solution which is dual to yours (i.e., testing a data item against every set) and that is using a binary tree where each node tests whether a specific item is included or not.
For instance if you had the sets A = { 2, 3 } and B = { 4 } and C = { 1, 3 } you'd have the following tree
_NOT_HAVE_[1]___HAVE____
| |
_____[2]_____ _____[2]_____
| | | |
__[3]__ __[3]__ __[3]__ __[3]__
| | | | | | | |
[4] [4] [4] [4] [4] [4] [4] [4]
/ \ / \ / \ / \ / \ / \ / \ / \
. B . B . B . B B C B A A A A
C B C B
C
After making the tree, you'd simply need to make 50 comparisons---or how ever many items you can have in a set.
For instance, for { 1, 4 }, you branch through the tree : right (the set has 1), left (doesn't have 2), left, right, and you get [ B ], meaning only set B is included in { 1, 4 }.
This is basically called a "Binary Decision Diagram". If you are offended by the redundancy in the nodes (as you should be, because 2^50 is a lot of nodes...) then you should consider the reduced form, which is called a "Reduced, Ordered Binary Decision Diagram" and is a commonly used data-structure. In this version, nodes are merged when they are redundant, and you no longer have a binary tree, but a directed acyclic graph.
The Wikipedia page on ROBBDs can provide you with more information, as well as links to libraries which implement this data-structure for various languages.
I can't prove it, but I'm fairly certain that there is no solution that can easily beat the O(n) bound. Your problem is "too general": every set has m = 50 properties (namely, property k is that it contains the number k) and the point is that all these properties are independent of each other. There aren't any clever combinations of properties that can predict the presence of other properties. Sorting doesn't work because the problem is very symmetric, any permutation of your 50 numbers will give the same problem but screw up any kind of ordering. Unless your input has a hidden structure, you're out of luck.
However, there is some room for speed / memory tradeoffs. Namely, you can precompute the answers for small queries. Let Q be a query set, and supersets(Q) be the collection of sets that contain Q, i.e. the solution to your problem. Then, your problem has the following key property
Q ⊆ P => supersets(Q) ⊇ supersets(P)
In other words, the results for P = {1,3,4} are a subcollection of the results for Q = {1,3}.
Now, precompute all answers for small queries. For demonstration, let's take all queries of size <= 3. You'll get a table
supersets({1})
supersets({2})
...
supersets({50})
supersets({1,2})
supersets({2,3})
...
supersets({1,2,3})
supersets({1,2,4})
...
supersets({48,49,50})
with O(m^3) entries. To compute, say, supersets({1,2,3,4}), you look up superset({1,2,3}) and run your linear algorithm on this collection. The point is that on average, superset({1,2,3}) will not contain the full n = 50,000 elements, but only a fraction n/2^3 = 6250 of those, giving an 8-fold increase in speed.
(This is a generalization of the "reverse index" method that other answers suggested.)
Depending on your data set, memory use will be rather terrible, though. But you might be able to omit some rows or speed up the algorithm by noting that a query like {1,2,3,4} can be calculated from several different precomputed answers, like supersets({1,2,3}) and supersets({1,2,4}), and you'll use the smallest of these.
If you're going to improve performance, you're going to have to do something fancy to reduce the number of set comparisons you make.
Maybe you can partition the data items so that you have all those where 1 is the smallest element in one group, and all those where 2 is the smallest item in another group, and so on.
When it comes to searching, you find the smallest value in the search set, and look at the group where that value is present.
Or, perhaps, group them into 50 groups by 'this data item contains N' for N = 1..50.
When it comes to searching, you find the size of each group that holds each element of the set, and then search just the smallest group.
The concern with this - especially the latter - is that the overhead of reducing the search time might outweigh the performance benefit from the reduced search space.
You could use inverted index of your data items. For your example
1 {1, 2, 4, 7, 8, 12, 18, 23, 29}
2 {3, 4, 6, 7, 15, 23, 34, 38}
3 {4, 7, 12, 18}
4 {1, 4, 7, 12, 13, 14, 15, 16, 17, 18}
5 {2, 4, 6, 7, 13, 15}
the inverted index will be
1: {1, 4}
2: {1, 5}
3: {2}
4: {1, 2, 3, 4, 5}
5: {}
6: {2, 5}
...
So, for any particular set {x_0, x_1, ..., x_i} you need to intersect sets for x_0, x_1 and others. For example, for the set {2,3,4} you need to intersect {1,5} with {2} and with {1,2,3,4,5}. Because you could have all your sets in inverted index sorted, you could intersect sets in min of lengths of sets that are to be intersected.
Here could be an issue, if you have very 'popular' items (as 4 in our example) with huge index set.
Some words about intersecting. You could use sorted lists in inverted index, and intersect sets in pairs (in increasing length order). Or as you have no more than 50K items, you could use compressed bit sets (about 6Kb for every number, fewer for sparse bit sets, about 50 numbers, not so greedily), and intersect bit sets bitwise. For sparse bit sets that will be efficiently, I think.
A possible way to divvy up the list of bitmaps, would be to create an array of (Compiled Nibble Indicators)
Let's say one of your 64 bit bitmaps has the bit 0 to bit 8 set.
In hex we can look at it as 0x000000000000001F
Now, let's transform that into a simpler and smaller representation.
Each 4 bit Nibble, either has at least one bit set, or not.
If it does, we represent it as a 1, if not we represent it as a 0.
So the hex value reduces to bit pattern 0000000000000011, as the right hand 2 nibbles have are the only ones that have bits in them. Create an array, that holds 65536 values, and use them as a head of linked lists, or set of large arrays....
Compile each of your bit maps, into it's compact CNI. Add it to the correct list, until all of the lists have been compiled.
Then take your needle. Compile it into its CNI form. Use that to value, to subscript to the head of the list. All bitmaps in that list have a possibility of being a match.
All bitmaps in the other lists can not match.
That is a way to divvy them up.
Now in practice, I doubt a linked list would meet your performance requirements.
If you write a function to compile a bit map to CNI, you could use it as a basis to sort your array by the CNI. Then have your array of 65536 heads, simply subscript into the original array as the start of a range.
Another technique would be to just compile a part of the 64 bit bitmap, so you have fewer heads. Analysis of your patterns should give you an idea of what nibbles are most effective in partitioning them up.
Good luck to you, and please let us know what you finally end up doing.
Evil.
The index of the sets that match the search criterion resemble the sets themselves. Instead of having the unique indexes less than 50, we have unique indexes less than 50000. Since you don't mind using a bit of memory, you can precompute matching sets in a 50 element array of 50000 bit integers. Then you index into the precomputed matches and basically just do your ((set & data) == set) but on the 50000 bit numbers which represent the matching sets. Here's what I mean.
#include <iostream>
enum
{
max_sets = 50000, // should be >= 64
num_boxes = max_sets / 64 + 1,
max_entry = 50
};
uint64_t sets_containing[max_entry][num_boxes];
#define _(x) (uint64_t(1) << x)
uint64_t sets[] =
{
_(1) | _(2) | _(4) | _(7) | _(8) | _(12) | _(18) | _(23) | _(29),
_(3) | _(4) | _(6) | _(7) | _(15) | _(23) | _(34) | _(38),
_(4) | _(7) | _(12) | _(18),
_(1) | _(4) | _(7) | _(12) | _(13) | _(14) | _(15) | _(16) | _(17) | _(18),
_(2) | _(4) | _(6) | _(7) | _(13) | _(15),
0,
};
void big_and_equals(uint64_t lhs[num_boxes], uint64_t rhs[num_boxes])
{
static int comparison_counter = 0;
for (int i = 0; i < num_boxes; ++i, ++comparison_counter)
{
lhs[i] &= rhs[i];
}
std::cout
<< "performed "
<< comparison_counter
<< " comparisons"
<< std::endl;
}
int main()
{
// Precompute matches
memset(sets_containing, 0, sizeof(uint64_t) * max_entry * num_boxes);
int set_number = 0;
for (uint64_t* p = &sets[0]; *p; ++p, ++set_number)
{
int entry = 0;
for (uint64_t set = *p; set; set >>= 1, ++entry)
{
if (set & 1)
{
std::cout
<< "sets_containing["
<< entry
<< "]["
<< (set_number / 64)
<< "] gets bit "
<< set_number % 64
<< std::endl;
uint64_t& flag_location =
sets_containing[entry][set_number / 64];
flag_location |= _(set_number % 64);
}
}
}
// Perform search for a key
int key[] = {4, 7, 12, 18};
uint64_t answer[num_boxes];
memset(answer, 0xff, sizeof(uint64_t) * num_boxes);
for (int i = 0; i < sizeof(key) / sizeof(key[0]); ++i)
{
big_and_equals(answer, sets_containing[key[i]]);
}
// Display the matches
for (int set_number = 0; set_number < max_sets; ++set_number)
{
if (answer[set_number / 64] & _(set_number % 64))
{
std::cout
<< "set "
<< set_number
<< " matches"
<< std::endl;
}
}
return 0;
}
Running this program yields:
sets_containing[1][0] gets bit 0
sets_containing[2][0] gets bit 0
sets_containing[4][0] gets bit 0
sets_containing[7][0] gets bit 0
sets_containing[8][0] gets bit 0
sets_containing[12][0] gets bit 0
sets_containing[18][0] gets bit 0
sets_containing[23][0] gets bit 0
sets_containing[29][0] gets bit 0
sets_containing[3][0] gets bit 1
sets_containing[4][0] gets bit 1
sets_containing[6][0] gets bit 1
sets_containing[7][0] gets bit 1
sets_containing[15][0] gets bit 1
sets_containing[23][0] gets bit 1
sets_containing[34][0] gets bit 1
sets_containing[38][0] gets bit 1
sets_containing[4][0] gets bit 2
sets_containing[7][0] gets bit 2
sets_containing[12][0] gets bit 2
sets_containing[18][0] gets bit 2
sets_containing[1][0] gets bit 3
sets_containing[4][0] gets bit 3
sets_containing[7][0] gets bit 3
sets_containing[12][0] gets bit 3
sets_containing[13][0] gets bit 3
sets_containing[14][0] gets bit 3
sets_containing[15][0] gets bit 3
sets_containing[16][0] gets bit 3
sets_containing[17][0] gets bit 3
sets_containing[18][0] gets bit 3
sets_containing[2][0] gets bit 4
sets_containing[4][0] gets bit 4
sets_containing[6][0] gets bit 4
sets_containing[7][0] gets bit 4
sets_containing[13][0] gets bit 4
sets_containing[15][0] gets bit 4
performed 782 comparisons
performed 1564 comparisons
performed 2346 comparisons
performed 3128 comparisons
set 0 matches
set 2 matches
set 3 matches
3128 uint64_t comparisons beats 50000 comparisons so you win. Even in the worst case, which would be a key which has all 50 items, you only have to do num_boxes * max_entry comparisons which in this case is 39100. Still better than 50000.
Since the numbers are less than 50, you could build a one-to-one hash using a 64-bit integer and then use bitwise operations to test the sets in O(1) time. The hash creation would also be O(1). I think either an XOR followed by a test for zero or an AND followed by a test for equality would work. (If I understood the problem correctly.)
Put your sets into an array (not a linked list) and SORT THEM. The sorting criteria can be either 1) the number of elements in the set (number of 1-bits in the set representation), or 2) the lowest element in the set. For example, let A={7, 10, 16} and B={11, 17}. Then B<A under criterion 1), and A<B under criterion 2). Sorting is O(n log n), but I assume that you can afford some preprocessing time, i.e., that the search structure is static.
When a new data item arrives, you can use binary search (logarithmic time) to find the starting candidate set in the array. Then you search linearly through the array and test the data item against the set in the array until the data item becomes "greater" than the set.
You should choose your sorting criterion based on the spread of your sets. If all sets have 0 as their lowest element, you shouldn't choose criterion 2). Vice-versa, if the distribution of set cardinalities is not uniform, you shouldn't choose criterion 1).
Yet another, more robust, sorting criterion would be to compute the span of elements in each set, and sort them according to that. For example, the lowest element in set A is 7, and highest is 16, so you would encode its span as 0x1007; similarly the B's span would be 0x110B. Sort the sets according to the "span code" and again use binary search to find all sets with the same "span code" as your data item.
Computing the "span code" is slow in ordinary C, but it can be done fast if you resort to assembly -- most CPUs have instructions that find the most/least significant set bit.
This is not a real answer more an observation: this problem looks like it could be efficiently parallellized or even distributed, which would at least reduce the running time to O(n / number of cores)
You can build a reverse index of "haystack" lists that contain each element:
std::set<int> needle; // {4, 7, 12, 18}
std::vector<std::set<int>> haystacks;
// A list of your each of your data sets:
// 1 {1, 2, 4, 7, 8, 12, 18, 23, 29}
// 2 {3, 4, 6, 7, 15, 23, 34, 38}
// 3 {4, 7, 12, 18}
// 4 {1, 4, 7, 12, 13, 14, 15, 16, 17, 18}
// 5 {2, 4, 6, 7, 13,
std::hash_map[int, set<int>> element_haystacks;
// element_haystacks maps each integer to the sets that contain it
// (the key is the integers from the haystacks sets, and
// the set values are the index into the 'haystacks' vector):
// 1 -> {1, 4} Element 1 is in sets 1 and 4.
// 2 -> {1, 5} Element 2 is in sets 2 and 4.
// 3 -> {2} Element 3 is in set 3.
// 4 -> {1, 2, 3, 4, 5} Element 4 is in sets 1 through 5.
std::set<int> answer_sets; // The list of haystack sets that contain your set.
for (set<int>::const_iterator it = needle.begin(); it != needle.end(); ++it) {
const std::set<int> &new_answer = element_haystacks[i];
std::set<int> existing_answer;
std::swap(existing_answer, answer_sets);
// Remove all answers that don't occur in the new element list.
std::set_intersection(existing_answer.begin(), existing_answer.end(),
new_answer.begin(), new_answer.end(),
inserter(answer_sets, answer_sets.begin()));
if (answer_sets.empty()) break; // No matches :(
}
// answer_sets now lists the haystack_ids that include all your needle elements.
for (int i = 0; i < answer_sets.size(); ++i) {
cout << "set: " << element_haystacks[answer_sets[i]];
}
If I'm not mistaken, this will have a max runtime of O(k*m), where is the avg number of sets that an integer belongs to and m is the avg size of the needle set (<50). Unfortunately, it'll have a significant memory overhead due to building the reverse mapping (element_haystacks).
I'm sure you could improve this a bit if you stored sorted vectors instead of sets and element_haystacks could be a 50 element vector instead of a hash_map.
I'm surprised no one has mentioned that the STL contains an algorithm to handle this sort of thing for you. Hence, you should use includes. As it describes it performs at most 2*(N+M)-1 comparisons for a worst case performance of O(M+N).
Hence:
bool isContained = includes( myVector.begin(), myVector.end(), another.begin(), another.end() );
if you're needing O( log N ) time, I'll have to yield to the other responders.
Another idea is to completely prehunt your elephants.
Setup
Create a 64 bit X 50,000 element bit array.
Analyze your search set, and set the corresponding bits in each row.
Save the bit map to disk, so it can be reloaded as needed.
Searching
Load the element bit array into memory.
Create a bit map array, 1 X 50000. Set all of the values to 1. This is the search bit array
Take your needle, and walk though each value. Use it as a subscript into the element bit array. Take the corresponding bit array, then AND it into the search array.
Do that for all values in your needle, and your search bit array, will hold a 1,
for every matching element.
Reconstruct
Walk through the search bit array, and for each 1, you can use the element bit array, to reconstruct the original values.
How many data items do you have? Are they really all unique? Could you cache popular data items, or use a bucket/radix sort before the run to group repeated items together?
Here is an indexing approach:
1) Divide the 50-bit field into e.g. 10 5-bit sub-fields. If you really have 50K sets then 3 17-bit chunks might be nearer the mark.
2) For each set, choose a single subfield. A good choice is the sub-field where that set has the most bits set, with ties broken almost arbitrarily - e.g. use the leftmost such sub-field.
3) For each possible bit-pattern in each sub-field note down the list of sets which are allocated to that sub-field and match that pattern, considering only the sub-field.
4) Given a new data item, divide it into its 5-bit chunks and look each up in its own lookup table to get a list of sets to test against. If your data is completely random you get a factor of two speedup or more, depending on how many bits are set in the densest sub-field of each set. If an adversary gets to make up random data for you, perhaps they find data items that almost but not quite match loads of sets and you don't do very well at all.
Possibly there is scope for taking advantage of any structure in your sets, by numbering bits so that sets tend to have two or more bits in their best sub-field - e.g. do cluster analysis on the bits, treating them as similar if they tend to appear together in sets. Or if you can predict patterns in the data items, alter the allocation of sets to sub-fields in step(2) to reduce the number of expected false matches.
Addition:
How many tables would need to have to guarantee that any 2 bits always fall into the same table? If you look at the combinatorial definition in http://en.wikipedia.org/wiki/Projective_plane, you can see that there is a way to extract collections of 7 bits from 57 (=1 + 7 + 49) bits in 57 different ways so that for any two bits at least one collection contains both of them. Probably not very useful, but it's still an answer.