I got a small problem.. Currently im writing a small tool which connects to a Server, downloads some blowfish encrypted Data into a char Array (char buffer[512]) and saves that Array into a file (fstream ios::binary).
Then later i reopen that file (ifstream ios::binary) and send it back. This time i read it into a std::string and not a char[].
My Problem now is that the Data is corrupted after i send them back.
What would be the best way to acomblish this? Is it safe to use char[] and std::string? Or does that somehow corrupt the data? (i guess so).
If you need some code examples i can provide them.
Thanks!
##########################
Okay a quick update to give you more details.
Im writing a Tool for a game (on OS X), its seperated in two parts:
Retrieve the encrypted Data from Game Servers and save them.
Send the saved encrypted Data back to the local Game
Problem: When the saved Data arrives the local Game, the Game just spits out that the Data cant be decrypted => data corruption.
Here's the code for 1. (Its just poc for selftesting purpose):
mode == 0 => no gameDataChunk there
mode == 1 => gameDataChunk there, need to write
mode == 2 => keyFrame there
*/
std::string socketRead(int socket, int mode, std::string num_chunk, std::string num_key, std::string game_id){
std::string response = "";
char buffer[512];
int bytes;
while((bytes = read(socket, buffer, 511)) > 0){
std::string temp(buffer, bytes);
//std::cout << temp;
response = response + temp;
if(!(temp.find("HTTP/1.1 200")<1800000)){
if(temp.find("HTTP/1.1")<1800000){
std::cout << std::endl << std::endl << "GOT NOT A 200 !!!" << std::endl << std::endl;
std::cout << temp << std::endl << std::endl;
return "EXIT";
}
}
if(mode == 1){
if(temp.find("\r\n\r\n")<1800000){
write_gameDataChunk(buffer, 1, temp.find("\r\n\r\n")+4, num_chunk, game_id);
}
else{
write_gameDataChunk(buffer, 0, 0, num_chunk, game_id);
}
}
else if(mode == 2){
if(temp.find("\r\n\r\n")<1800000){
write_keyFrame(buffer, 1, temp.find("\r\n\r\n")+4, num_key, game_id);
}
else{
write_keyFrame(buffer, 0,0, num_key, game_id);
}
}
bzero(buffer, 512);
}
return response;
}
void write_gameDataChunk(char buffer[], int mode, int index, std::string num, std::string game_id){
std::fstream write;
char local_buffer[512];
if(mode == 0){
//complete bin data
write.open("/tmp/" + game_id + "/" + num + "/game_dataChunk.bin", std::ios::out | std::ios::binary | std::ios::app);
write << buffer;
}
else if(mode == 1){
//partial bin data
std::string command = "mkdir /tmp/" + game_id + "/" + num;
std::system(command.c_str());
write.open("/tmp/" + game_id + "/" + num + "/game_dataChunk.bin", std::ios::out | std::ios::binary);
strcpy(local_buffer, buffer+index);
write << local_buffer;
}
write.close();
}
Here is the code for 2.:
std::string get_DataChunk(std::string game_id, int chunkId){
std::ifstream read;
std::string dummy;
std::string command;
std::string result = "";
command = "/tmp/" + game_id + "/" + std::to_string(chunkId) + "/game_dataChunk.bin";
read.open(command.c_str(), std::ios::in | std::ios::binary);
while(getline(read,dummy)){
result = result + dummy;
}
read.close();
return result;
}
This is all testing code for myself and not finished, so its kinda ugly... but i hope you will understand what i mean. I did not often had to deal with binary Data send over Sockets..
std::string is not meant to deal with binary. It is meant to deal with strings.
What you want is a std::vector<char> which doesn't care about what values you are storing.
Any '\0' in your binary data will stop construct the std::string. So the std::string().size() only count to the character before 1st '\0'. Use std::vector<unsigned char> instead.
By the way, what does 1800000 mean? I know that HTTP/1.1 200 is the characters you received at the earliest.(poor english)
Related
I have a file which is read in as binary. It has some basic plain text at the top and image data after. Due to a process I cannot control I need to ensure that a certain string of characters is not present in the plain text representation of the data. Meaning that if you opened the image data as plain text and the character sequence was present that is a problem. My thought was to read in the data, convert to hex, and replace the the instances of that sequence. The process would then be reversed to restore the file.
std::ifstream stream;
std::stringstream ss;
if (stream.bad())
do.something();
length = stream.rdbuf()->pubseekoff(0, std::ios_base::end);
data = new char[length];
stream.rdbuf()->pubseekoff(0, std::ios_base::beg);
stream.read(data, length);
stream.close();
ss << std::hex;
for (int i = 0; i < length; ++i)
ss << std::setw(2) << std::setfill('0') << (int)data[i];
const std::string& tmp = ss.str();
const char* cstr = tmp.c_str();
CString hexStr(cstr);
This reads the top portion of the file fine but introduces 'FF' characters into the rest of the data. My c++ isn't great so I'm looking for some help as to why and how to correctly do this. The rest looks like:
hexStr.Replace(character sequence to replace, replacement);
std::ofstream datafile(pathToFile, std::ios_base::binary | std::ios_base::out);
char buf[3];
buf[2] = 0;
std::stringstream input(hexStr.GetString());
input.flags(std::ios_base::hex);
while (input)
{
input >> buf[0] >> buf[1];
long val = strtol(buf, nullptr, 16);
datafile << static_cast<unsigned char>(val & 0xff);
}
I have been experiencing a bug for the past day that I have not been able to solve.
I have my first method which is for saving player data:
bool Player::savePlayerData() {
ofstream writeFile(getName() + ".bin", ios::out | ios::binary | ios::trunc);
string writeData;
writeData = formatEntityData() + "<" + formatLocationData() + "<" + formatInventory();
writeFile.write(writeData.c_str(), writeData.length() + 1);
writeFile.close();
return true;
}
Note: Assume that getName(), formatEntityData(), formatLocationData(), and formatInventory() return strings and are functional.
Then I have my load player data method:
bool Player::loadPlayerData(string name) {
ifstream readFile(name + ".bin", ios::in | ios::binary | ios::_Nocreate);
if (readFile.good() && readFile.is_open()) {
string data;
getline(readFile, data, '\0');
vector<string> str = split(data, '<');
parseEntityData(str.at(0));
parseLocationData(str.at(1));
parseInventory(str.at(2));
readFile.close();
return true;
}
readFile.close();
return false;
}
Note: Assume that parseEntityData(), parseLocationData(), parseInventory() have string param, void returns and are functional
Note: Assume that split(string, char) takes in a string with a delim. char and splits into vector correctly
So, here is what I am trying to accomplish (for purposes of simplicity lets assume getName() return "luke"):
•Create luke.bin
•Save string to luke.bin in binary
•Load data from luke.bin in form of a string
When I run the program is not properly reading the player data. Instead it is returning as if nothing is in the file. What am I doing wrong? Any tips, ideas, or thoughts would be greatly appreciated.
Code on brothers!
Typically when you open a binary file in notepad++ it gives seemingly
random characters
It depends on data. The string "Hell world" is the same in binary or text. Numbers will appear as text if they are text formatted.
Example of text format:
fout << 1234 << std::endl; //saved as "1234"
Example of binary data:
int i = 1234;
fout.write(&i, sizeof(i)); //saved as 2 bytes, big-endian or little endian binary
ios::binary stops translation of new line characters.
When writing to file, put the exact size:
writeFile.write(writeData.c_str(), writeData.length());
When reading the file, getline(fin, data, '\0'); will stop when it reaches zero or end of file. You should use EOF instead of zero. Better yet, use this method:
std::ifstream f(filename, ios::binary);
if (f.good())
{
f.seekg(0, ios::end);
size_t filesize = (size_t)f.tellg();
f.seekg(0);
std::string data(filesize, 0);
f.read(&data[0], filesize);
cout << data << endl;
return true;
}
return false;
I'm new to the forum, but not to this website. I've been searching for weeks on how to process a large data file quickly using C++ 11. I'm trying to have a function with a member that will capture the trace file name, open and process the data. The trace file contains 2 million lines of data, and each line is structured with a read/write operation and a hex address:
r abcdef123456
However, with a file having that much data, I need to read in and parse those 2 values quickly. My first attempt to read the file was the following:
void getTraceData(string filename)
{
ifstream inputfile;
string file_str;
vector<string> op, addr;
// Open input file
inputfile.open(filename.c_str());
cout << "Opening file for reading: " << filename << endl;
// Determine if file opened successfully
if(inputfile.fail())
{
cout << "Text file failed to open." << endl;
cout << "Please check file name and path." << endl;
exit(1);
}
// Retrieve and store address values and operations
if(inputfile.is_open())
{
cout << "Text file opened successfully." << endl;
while(inputfile >> file_str)
{
if((file_str == "r") || (file_str == "w"))
{
op.push_back(file_str);
}
else
{
addr.push_back(file_str);
}
}
}
inputfile.close();
cout << "File closed." << endl;
}
It worked, it ran, and read in the file. Unfortunately, it took the program 8 minutes to run and read the file. I modified the first program to the second program, to try and read the file in faster. It did, reading the file into a buffer in a fraction of a second versus 8 mins. using ifstream:
void getTraceData()
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
}
But when I added the parsing portion of the code, to get each line, and parsing the buffer contents line-by-line to store the two values in two separate variables, the program silently exits at the while-loop containing getline from the buffer:
void getTraceData(string filename)
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
string op, addr, line;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
// Setup stream buffer
const int maxline = 20;
char* lbuffer;
stringstream ss;
// Parse buffer data line-by-line
while(ss.getline(lbuffer, length))
{
while(getline(ss, line))
{
ss >> op >> addr;
}
ss.ignore( strlen(lbuffer));
}
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
delete[] lbuffer;
}
I was wondering, once my file is read into a buffer, how do I retrieve it and store it into variables? For added value, my benchmark time is under 2 mins. to read and process the data file. But right now, I'm just focused on the input file, and not the rest of my program or the machine it runs on (the code is portable to other machines). The language is C++ 11 and the OS is a Linux computer. Sorry for the long posting.
Your stringstream ss is not associated to fbuffer at all. You are trying to getline from an empty stringstream, thus nothing happens. Try this:
string inputedString(fbuffer);
istringstream ss(fbuffer);
And before ss.getline(lbuffer, length), please allocate memory for lbuffer.
Actually you can directly read your file into a string to avoid the copy construction. Check this Reading directly from an std::istream into an std::string .
Last but not least, since your vector is quite large, you'd better reserve enough space for it before push_back the items one by one. When a vector reaches its capacity, attempt to push_back another item into it will result in reallocation and copy of all previous items in order to ensure continuous storage. Millions of items will make that happen quite a few times.
I am quite new to C++ and am trying to work out how to write a record in the format of this structure below to a text file:
struct user {
int id;
char username [20];
char password [20];
char name [20];
char email [30];
int telephone;
char address [70];
int level;
};
So far, I'm able to write to it fine but without an incremented id number as I don't know how to work out the number of records so the file looks something like this after I've written the data to the file.
1 Nick pass Nick email tele address 1
1 user pass name email tele address 1
1 test test test test test test 1
1 user pass Nick email tele addy 1
1 nbao pass Nick email tele 207 1
Using the following code:
ofstream outFile;
outFile.open("users.dat", ios::app);
// User input of data here
outFile << "\n" << 1 << " " << username << " " << password << " " << name << " "
<< email << " " << telephone << " " << address << " " << 1;
cout << "\nUser added successfully\n\n";
outFile.close();
So, how can I increment the value for each record on insertion and how then target a specific record in the file?
EDIT: I've got as far as being able to display each line:
if (inFile.is_open())
{
while(!inFile.eof())
{
cout<<endl;
getline(inFile,line);
cout<<line<<endl;
}
inFile.close();
}
What you have so far is not bad, except that it cannot handle cases where there is space in your strings (for example in address!)
What you are trying to do is write a very basic data base. You require three operations that need to be implemented separately (although intertwining them may give you better performance in certain cases, but I'm sure that's not your concern here).
Insert: You already have this implemented. Only thing you might want to change is the " " to "\n". This way, every field of the struct is in a new line and your problem with spaces are resolved. Later when reading, you need to read line by line
Search: To search, you need to open the file, read struct by struct (which itself consists of reading many lines corresponding to your struct fields) and identifying the entities of your interest. What to do with them is another issue, but simplest case would be to return the list of matching entities in an array (or vector).
Delete: This is similar to search, except you have to rewrite the file. What you do is basically, again read struct by struct, see which ones match your criteria of deletion. You ignore those that match, and write (like the insert part) the rest to another file. Afterwards, you can replace the original file with the new file.
Here is a pseudo-code:
Write-entity(user &u, ofstream &fout)
fout << u.id << endl
<< u.username << endl
<< u.password << endl
<< ...
Read-entity(user &u, ifstream &fin)
char ignore_new_line
fin >> u.id >> ignore_new_line
fin.getline(u.username, 20);
fin.getline(u.password, 20);
...
if end of file
return fail
Insert(user &u)
ofstream fout("db.dat");
Write-entity(u, fout);
fout.close();
Search(char *username) /* for example */
ifstream fin("db.dat");
user u;
vector<user> results;
while (Read-entity(u))
if (strcmp(username, u.username) == 0)
results.push_back(u);
fin.close();
return results;
Delete(int level) /* for example */
ifstream fin("db.dat");
ofstream fout("db_temp.dat");
user u;
while (Read-entity(u))
if (level != u.level)
Write-entity(u, fout);
fin.close();
fout.close();
copy "db_temp.dat" to "db.dat"
Side note: It's a good idea to place the \n after data has been written (so that your text file would end in a new line)
Using typical methods at least you will need to use fix size records if you want to have random access when reading the file so say you have 5 characters for name it will be stored as
bob\0\0
or whatever else you use to pad, this way you can index with record number * record size.
To increment the index you in the way you are doing you will need to the read the file to find the high existing index and increment it. Or you can load the file into memory and append the new record and write the file back
std::vector<user> users=read_dat("file.dat");
user user_=get_from_input();
users.push_back(user_);
then write the file back
std::ofstream file("file.dat");
for(size_t i=0; i!=users.size(); ++i) {
file << users.at(i);
//you will need to implement the stream extractor to do this easily
}
I suggest to wrap the file handler into a Class, and then overload the operator >> and << for your struct, with this was you will control the in and out.
For instance
struct User{
...
};
typedef std::vector<User> UserConT;
struct MyDataFile
{
ofstream outFile;
UserConT User_container;
MyDataFile(std::string const&); //
MyDataFile& operator<< (User const& user); // Implement and/or process the record before to write
MyDataFile& operator>> (UserConT & user); // Implement the extraction/parse and insert into container
MyDataFile& operator<< (UserConT const & user); //Implement extraction/parse and insert into ofstream
};
MyDataFile& MyDataFile::operator<< (User const& user)
{
static unsigned myIdRecord=User_container.size();
myIdRecord++;
outFile << user.id+myIdRecord << ....;
return *this;
}
int main()
{
MydataFile file("data.dat");
UserConT myUser;
User a;
//... you could manage a single record
a.name="pepe";
...
file<<a;
..//
}
A .Dat file is normally a simple text file itself that can be opened with notepad . So , you can simply read the Last Line of the file , read it , extract the first character , convert it into integer . THen increment the value and be done .
Some sample code here :
#include <iostream.h>
#include <fstream.h>
using namespace std;
int main(int argc, char *argv[])
{
ifstream in("test.txt");
if(!in) {
cout << "Cannot open input file.\n";
return 1;
}
char str[255];
while(in) {
in.getline(str, 255); // delim defaults to '\n'
//if(in) cout << str << endl;
}
// Now str contains the last line ,
if ((str[0] >=48) || ( str[0] <=57))
{
int i = atoi(str[0]);
i++;
}
//i contains the latest value , do your operation now
in.close();
return 0;
}
Assuming your file format doesn't not need to be human readable.
You can write the struct out to file such as.
outFile.open("users.dat", ios::app | ios::binary);
user someValue = {};
outFile.write( (char*)&someValue, sizeof(user) );
int nIndex = 0;
user fetchValue = {};
ifstream inputFile.open("user.data", ios::binary);
inputFile.seekg (0, ios::end);
int itemCount = inputFile.tellg() / sizeof(user);
inputFile.seekg (0, ios::beg);
if( nIndex > -1 && nIndex < itemCount){
inputFile.seekg ( sizeof(user) * nIndex , ios::beg);
inputFile.read( (char*)&fetchValue, sizeof(user) );
}
The code that writes to the file is a member function of the user struct?
Otherwise I see no connection with between the output and the struct.
Possible things to do:
write the id member instead of 1
use a counter for id and increment it at each write
don't write the id and when reading use the line number as id
I am creating a very simple webserver, as practice in C++ and sockets. I use OSX.
The code sample is from inside the while(1) loop, a connection has been made and I am starting to process the header. This code works for all text-files but it dosn't work with images. And I figure that I can't use the same method to read text-files and images since images isn't separeted with lines. But how do I read the image data to send through the socket? I might not even be able to use a string, do I have to use char*?
string strFile = "htdocs" + getFileFromHeader(httpRequestHeader);
string strExt = getFileExtension(strFile);
string httpContent = "";
ifstream fileIn(strFile.c_str(), ios::in); // <-- do I have to use ios::binary ?
if(!fileIn)
{
// 404
cout << "File could not be opened" << endl;
httpContent = httpHeader404;
}
else
{
// 200
string contentType = getContentType(strExt);
cout << "Sending " << strFile << " -- " << contentType << endl;
string textInFile = "";
while (fileIn.good())
{
getline (fileIn, textInFile); // replace with what?
httpContent = httpContent + textInFile + "\n";
}
httpContent = httpHeader200 + newLine + contentType + newLine + newLine + httpContent;
}
// Sending httpContent through the socket
The question is about how to read the image data.
*EDIT 2011-05-19 *
So, this is an updated version of my code. The file have been opened with ios::binary, however, there are more problems.
httpContent = httpHeader200 + newLine + contentType + newLine + newLine;
char* fileContents = (char*)httpContent.c_str();
char a[1];
int i = 0;
while(!fileIn.eof())
{
fileIn.read(a, 1);
std::size_t len = std::strlen (fileContents);
char *ret = new char[len + 2];
std::strcpy ( ret, fileContents );
ret[len] = a[0];
ret[len + 1] = '\0';
fileContents = ret;
cout << fileContents << endl << endl << endl;
delete [] ret;
i++;
}
The problem is that is seems that the char * fileContents empty itself every ~240 chars. How can that be? Is there some sort of limit to some of theese functions that they only accept certain length?
Open the file for binary read, store the data in a char* array large enough, then send that array.
As #Blackbear said, but don't forget to send the corresponding HTML-Headers like contentEncoding, transferEncoding, etc. For simplicity try to send the binary Data of the image encoded in base64.