find a frequency of words in text file - c++

Question

Please can anybody help me? I'm a beginner and I have a hard assignment.
I need to write a c++ program that does the following :
Ask the user to enter two text file the first one contains the text file, the second one contains a list of words in one column Regardless of their number like this:
//output
Enter the text file: text_file.txt
Enter the keywords file: keywords_file.txt
Search for the keywords from the keywords file in the text file
Find the appearance of each word like this :
system: 55 times
analysis: 21 times
Then output in new text file (ofstream)
This is my code it's coded properly ..but it asks the user to enter the words..I want it to take the words from the input text file ..and show the output on the text file (ofstream)
this is a part of keywords contents
//they are as a list in the original file
List item
model,
management,
e-commerce,
system,
cluster
infrastructure,
computer,
knowledge,
metadata,
process,
alter,
simulate,
stock,
inventory,
strategy,
plan,
historical,
deices,
exact,
Analyst,
break even point,
SWOT,
tactic,
develop,
prototype,
feasible,
Inferences,
busy,
cloud compute,
schema,
enterprise,
custom,
expert system,
structure,
data mine,
data warehouse,
organism,
data mart,
operate,
quality assure,
forecast,
report,
this is a part of the book contents
the circuit is characterised by long straights and chicanes. this means the cars’ engines
are at full throttle for over 75% of the lap, a higher percentage than most other circuits.
the track requires heavier-than-average braking over a given lap, as the cars repeatedly
decelerate at the end of some of the world’sfasTest straights for the slow chicanes.
the chicanes are lined by rugged kerbs. riding over these hard is crucial for fast laps.
the long straights require small wings for minimum drag. this means lower downforce,
resulting in lower grip on corners and under braking, and less stability over bumps.
the main high-speed corners lesmo 1, lesmo 2 and parabolica are all right turns.
parts of the circuit are surrounded by trees, which means leaves can be blown
onto the track.
#include <algorithm>
#include <fstream>
#include <iostream>
#include <string>
#include <unordered_map>
#include <vector>
using namespace std;
int main() {
ofstream ofs("output.txt");
ofs << "Keywords " << endl;
string kw_file;
cout << "enter the file name :" << " ";
getline(cin, kw_file);
ifstream keywordFile(kw_file.c_str());
string text_file;
cout << "enter the file name :" << " ";
getline(cin, text_file);
ifstream textFile(text_file.c_str());
if (!keywordFile.is_open() || !textFile.is_open()) {
std::cout << "Error in opening files\n";
return 1;
}
{//vector method
size_t i = 0;
std::vector<std::string> keywordVector;
std::string keyword;
while (keywordFile >> std::ws >> keyword) {
keywordVector.push_back(keyword);
}
std::vector<int> countVector(keywordVector.size());
std::string textWord;
while (textFile >> std::ws >> textWord) {
for (size_t i = 0; i < keywordVector.size(); ++i) {
if (keywordVector[i] == textWord) {
countVector[i]++;
}
}
}
for (size_t i = 0; i < keywordVector.size(); ++i) {
ofs << "The number of times [" << keywordVector[i] << "] appears in
textFile is " << countVector[i] << '\n';
}
}
keywordFile.clear(); textFile.clear();
keywordFile.seekg(0);
textFile.seekg(0);
}

Answer 1

Preliminary
Before even starting to solve this problem, let's think about what is actually required from this:
Read in the keywords from keywords_file.txt
Store these keywords in a data structure for later use.
Read in words from text_file.txt
Use the data structure from Step#2 to compare the words read in Step#3
Data structure simply refers to how your program stores the data that it works on. When you create an array such as
int arr[] = {1, 2, 3};
to later do some manipulations on, that too is a data structure.
Choosing the Data Structure for the Problem
From the above steps, reading in words from the file is a simple problem to solve. The most pressing problem is to figure out the data structure in Step#2.
Let's try to start with a basic one: vector. We are going to use vector instead of arrays like in the C-language primarily for 2 reasons:
Fixed arrays are simple to handle and are the preferred data structure when we have fixed data. But in this problem, we have dynamic data because the number of words in the keywords_file.txt could vary. It's much easier to use vectors as dynamic arrays rather than C-style dynamic arrays because it's much easier to manage memory with vector than with a plain array.
We are using C++ and not C, so let's try to use the data structures available in the Standard Library of C++.
If we choose a vector as the data structure to store the words from keywords_file.txt, then we would also need a data strcture to store the count of each word which would be the number of times it's found in text_file.txt. We can again use a vector for this. We could have some code like (pseudo-code only)
Read in a keyword(using a std::string) from keywords_file.txt
Store it in a vector (using the push_back() function) [Let's call this vector as keywordVector]
Repeat Step#1 and #2 until all the words from keywords_file.txt are read in and added to keywordVector.
Create another vector (let's call it countVector for storing the counts of the word which is of the same size as that of keywordVector.
Initialize all the values in countVector with 0.
Read in a word(again a std::string) from text_file.txt
Search for this word in the vector.
If the word is found in the vector, increment that word's count by increasing the count in countVector for that word, else ignore the word
Repeat Step#5 and Step#6 till all the words from text_file.txt are read in.
Note for Step#8, if word from text_file.txt matches with word at keywordVector[i], then increment the value of the corresponding index in the countVector i.e. ++countVector[i]
This should work for small programs and should solve the problem. Here's how the code would look like:
std::vector<std::string> keywordVector;//empty vector of strings
std::string keyword; //string to store each word read in from keywords_file.txt
while(keywordFile>>std::ws>>keyword) {//std::ws just ignores any whitespace
keywordVector.push_back(keyword);
}
//create vector of same size as keywordVector to keep track of count
std::vector<int> countVector(keywordVector.size());
std::string textWord;
while(textFile>>std::ws>>textWord) {
//Search through the keywordVector tocheck if any word matches
for(size_t i = 0; i < keywordVector.size(); ++i) {
if(keywordVector[i] == textWord) {
countVector[i]++;//Found a match, increment the count
}
}
}
Is that all there is?
No! The above method should work fine for small programs but things become problematic when the number of keywords becomes large and/or the number of words in text_file.txt becomes large.
Imagine this. Your keywords_file.txt becomes very large since it stores all the words in the English Oxford Dictionary and has around a million words. The first part of reading in the keywords would be fine since we require it anyway. But the next part is where the problems start. Imagine all the words in your text_file.txt were the word zebra. Now while searching for zebra in this list, you would have to go through that while loop every time for each word. If there are a billion words in your text_file.txt then you would end up doing a million iterations for each word that you read and the total iterations would be 1billion X 1 million = 1 quadrillion iterations (1 followed by 15 zeroes). This is too big. Your program would complete in a very long time. Assuming each iteration takes 1 nanosecond, then total time would be 1 million seconds (10^15* 10^-9) which is ((1000000/60)/60)/24 = around 11.5 days!! Surely that is not acceptable!
We realize that the basic problem is that we have to search the entire loop every time we read in a word from text_file.txt. Only if there was a way to just lookup the word in the keywordVector directly without having to iterate over the entire vector each and every time.
This is exactly what map data structure helps with. These data structures use a hash-function to quickly lookup things in a collection. They store what is called Key-Value pairs and you use the key to lookup its value.
e.g. if we have something like
{ 101: "Alice", 202: "Bob", 303: "Charlie"}
Then 101 is the key and Alice is the value of that key, 202 is the key, Bob is the value etc.
In the C++ STL there are 2 data structures that are build upon this concept std::unordered_map and std::map. As the names suggest, the first one stores the keys in no particular order but the second one stores them in a sorted way.
Given this very basic intro to maps, we can see how this might be helpful for our case. We don't need to store the words from keyword_file.txt in any particular order. So we can use std::unordered_map, and use the keyword as the key for this map. We'll store the number of times that word appears as the value of this key. Here's some code:
/* Create a map to store key-value pairs of
string and int, where each string is a keyword
from the keywords_file.txt
*/
std::unordered_map<std::string, int> keywordMap;
std::string keyword;
while(keywordFile>>std::ws>>keyword) {
//Initialize each word's count to 0
keywordMap[keyword] = 0;
}
std::string textWord;
while(textFile>>std::ws>>textWord) {
/*We do a find for the textWord in the map.
This find isn't a linear loop like thing (unlike vector)
but uses hashing to quickly look up if textWord exists in the
map or not
*/
if(keywordMap.find(textWord) != end(keywordMap)) {
//If it exists, then we can just directly increment the count
keywordMap[textWord]++;
}
}
Using our time calculations, this time around the lookup of a billion words from the text_file.txt would each take up only in the order of a few nano-seconds since the unordered_map.find() has an average case constant time complexity, unlike the earlier approach's linear complexity.
So for a billion words, it takes an order of a billion nano-seconds which is just 1 second! Imagine the drastic difference in the times. The earlier method took days and this takes seconds. Hashing is a very powerful concept and finds applications in a lot of problems.
Followup
Here's the full code, if you want to use it. This is a basic solution to finding the frequency of words in a file. Since you're beginning out in C++, I'd suggest you take your time to read in-depth into all the data structures used here and use this example to build upon your understanding. Also, if complexity is new to you, please do acquaint yourself with the topic.