Random selection of dictionary words - c++

Question

I want to select a number of random words from an array to make a total amount of 36 letters.
At first I tried to select a random word and add it after checking that it's not longer than the amount of free space we have. That was not efficient since the list would fill up and there would only be empty space left for a 2-3 letter word and it takes a long time to find such a short word.
So i decided to only choose six 6-letter words and I'm doing that by generating a random number and then incrementing it by 1 until we find a 6 letter word. It's pretty fast, but the words aren't really that random, often I get words that start from the same letter or only words that start with letters in sequence like a,b,c or x,y,z.
srand ( time(NULL) );
for(int i=0;i<6;i++)
{
randNumb = rand()%dictionary.size();
while(dictionary.at(randNumb).length() != 6)
{
randNumb++;
}
a << "/" << dictionary.at(randNumb) << "/";
}
I would like to choose words with different lengths but in favor of performance I'll settle with just the 6-letter words but then i would at least want them to be more randomly selected.

Answer 1

You should get a new random number instead of increasing the index. The way you do it, all the strings not matching your criteria "attract" more random numbers, and possibly lead to the following string to have a higher probability of being chosen.

Answer 2

The rand() function generates a number between 0 and RAND_MAX.
If RAND_MAX is defined as 32767, then you will not access elements in your dictionary (array?) with indices greater than that.
If you need to generate a random number greater than RAND_MAX, then think about summing the result of n calls of rand(), such that n * RAND_MAX >= dictionary.size(). The modulus of this result is then guaranteed to give an index that falls somewhere in the bounds of the entire dictionary.

Answer 3

Even if RAND_MAX is greater than dictionary.size(), using the % operator to select the index leads to a non-uniform distribution. The modulus will cause the early words to be selected more often than the later words (unless RAND_MAX + 1 is an integer multiple of dictionary.size()).
Consider a simple example: Assume your dictionary has 10 words, and RAND_MAX is 14. When rand() returns a value from 0 to 9, the corresponding word is chosen directly. But when rand() is 10 through 14, then one of the first five words will be chosen. So the first five words have twice the chance of being selected than the last five words.
A better way to map [0..RAND_MAX] to [0..dictionary.size()) is to use division:
assert(RAND_MAX + 1 >= dictionary.size());
randNumb = rand() * dictionary.size() / (RAND_MAX + 1);
But you have to be careful of integer overflow. If RAND_MAX * dictionary.size() is larger than you can represent in an integer, you'll need to use a larger data type. Some systems have a function like MulDiv for just this purpose. If you don't have something like MulDiv, you can convert to a floating point type and then truncate the result back to an integer:
double temp = static_cast<double>(rand()) * dictionary.size() / (RAND_MAX + 1);
randNumb = static_cast<int>(temp);
This is still an imperfect distribution, but the "hot" words will now be evenly distributed across the dictionary instead of clumping at the beginning.
The closer RAND_MAX + 1 is to an integer multiple of dictionary.size(), the better off you'll be. And if you can't be sure that it's close to an integer multiple, then you want RAND_MAX to be as large as possible relative to dictionary.size().
Since you don't have much control over RAND_MAX, you could consider tweaking dictionary.size(). For example, if you only want six-letter words, then why not strip all the others out of the dictionary?
std::vector<std::string> six_letter_words;
std::copy_if(dictionary.begin(), dictionary.end(),
std::back_inserter(six_letter_words),
[](const std::string &word){ return word.size() == 6; });
With the reduced set, we can use a more generic algorithm to select the words:
typedef std::vector<std::string> WordList;
// Returns true with the given probability, which should be 0.0 to 1.0.
bool Probably(double probability) {
return (static_cast<double>(std::rand()) / RAND_MAX) < probability;
}
// Selects n words from the dictionary using a normal distribution and
// copies them to target.
template <typename OutputIt>
OutputIt Select(int n, const WordList &dictionary, OutputIt target) {
double count = static_cast<double>(n);
for (std::size_t i = 0; count > 0.0 && i < dictionary.size(); ++i) {
if (Probably(count / (dictionary.size() - i))) {
*target++ = dictionary[i];
count -= 1.0;
}
}
return target;
}
The idea is to step through each word in the dictionary and select it with a probability of the number of words you need to pick divided by the number of words left to pick from. This works well, even if RAND_MAX is relatively small. Overall, though, it's much more computation than trying to randomly select indexes. Also note that this technique will never choose the same word more than once, where the index mapping technique could.
You call Select like this:
// Select six words from six_letter_words using a normal distribution.
WordList selected;
Select(6, six_letter_words, std::back_inserter(selected));
Also note that most implementations of rand() are pretty simplistic and may not give a good normal distribution to begin with.