algorithm to identify string matches with hashing and no false positives - c++

Question

I need to find if a given string matches strings in a list without having the strings in the list; basically I need to hash the strings and match only against the list of hashes. The problem is being sure that there are no false positives so that only exact matches will be found and any other set of characters is not found. This is of course easy with an actual list of strings, even a simple binary search will work, but I want an algorithm that works without the actual characters present (i.e. precalculated). A bloom filter doesn't have a guarantee that some arbitrary set of characters might not be matched.
Update: this is similar to storing only password hashes and then hashing an entered password which is then compared to the hash list to see if the password is one of them (admittedly not the usual use of a password). The reason for this requirement is to no have to ship the actual text, just the hashes.
Update 2: Is there another way to do this without a perfect hash function? I have hundreds of thousands of entries, finding a perfect hash is hard. Maybe something like a bloom filter but with a better guarantee?

Answer 1

A good cryptographic hash function (with sufficient bits) will make the probability of a false match extremely small; sufficiently small that brute force attacks are essentially impossible. Most security systems feel that such mechanisms are adequate.
If you want an absolute guarantee that no false positive is possible, then you'll actually need to include enough data to validate the input; that cannot be any shorter than the target strings (but it doesn't have to be any larger). In effect, you need to encrypt the target strings. Since the encryption key and the encrypted strings will both be visible, in order to avoid someone simply decrypting the encrypted strings you need to use an asymmetric cipher. Those are computationally expensive, but that might not be a problem for your environment.

Answer 2

Any perfect hash will do. Follow it up with a string compare to verify it is not a false positive.

Answer 3

Here's a "almost perfect" hashing algorithm:
MPQ Hash
It is used in the StarCraft save files. This algorithm is very efficient, and has a very low collision possibility (about 1:18889465931478580854784 on average).
Here is how this algorithm works.
1.Compute the three hashes (offset hash and two check hashes) and store them in variables.
2.Move to the entry of the offset hash
3.Is the entry unused? If so, stop the search and return 'file not found'.
4.Do the two check hashes match the check hashes of the file we're looking for? If so, stop the search and return the current entry.
5.Move to the next entry in the list, wrapping around to the beginning if we were on the last entry.
6.Is the entry we just moved to the same as the offset hash (did we look through the whole hash table?)? If so, stop the search and return 'file not found'.
7.Go back to step 3.
And here is the hashing and hash table function:
unsigned long HashString(char *lpszFileName, unsigned long dwHashType)
{
//lpszFileName is the string to be hashed.
//dwHashType will change the hash value according to hash mode.
//You can see how it's used in the beginning of GetHashTablePos().
unsigned char *key = (unsigned char *)lpszFileName;
unsigned long seed1 = 0x7FED7FED, seed2 = 0xEEEEEEEE;
int ch;
while(*key != 0)
{
ch = toupper(*key++); //Convert every character to upper case.
//dwHashType will change the hash value in different hashing modes.
//(Whether to calculate the position or to verify.)
seed1 = cryptTable[(dwHashType << 8) + ch] ^ (seed1 + seed2);
seed2 = ch + seed1 + seed2 + (seed2 << 5) + 3;
}
return seed1;
}
int GetHashTablePos(char *lpszString, MPQHASHTABLE *lpTable, int nTableSize)
{
const int HASH_OFFSET = 0, HASH_A = 1, HASH_B = 2;
//nHash controls where the hash value of the string should be stored in the hash table.
//nHashA and nHashB are used for verifying the match
int nHash = HashString(lpszString, HASH_OFFSET),
nHashA = HashString(lpszString, HASH_A),
nHashB = HashString(lpszString, HASH_B),
nHashStart = nHash % nTableSize, nHashPos = nHashStart;
while (lpTable[nHashPos].bExists)
{
if (lpTable[nHashPos].nHashA == nHashA && lpTable[nHashPos].nHashB == nHashB)
return nHashPos; //If found, return the entry index
else
nHashPos = (nHashPos + 1) % nTableSize; //Not found, move to next position
if (nHashPos == nHashStart)
break; //Reach the beginning of table, stop searching.
}
return -1; //Error value
}

Answer 4

You might consider putting together a bloom filter. That is basically a set of hash algorithms and hash tables which when taken as a group can give a very high probabilty of a correct match. It's not 100%, but you can get as close to it as you like.