I need to read file in parts ( for example by 4 bytes) and then increment numbers in files by one and then write back;
this part only fills in file by 1; How to increase this number on 1?
void Prepare()
{
//ifstream fileRead("\FILE", ios::in | ios::binary);
ofstream fileOut("\FILE.bin", ios::out | ios::binary);
int count = 10485760;
for (int i = 0; i < count-1; i++)
{
fileOut << 1;
}
fileOut.close();
}
If I understand your question, you need to read the file then write it out, changing the data. You can't really do it the way you've started.
There are two basic ways to do this. You can read the entire file into memory, then manipulate the memory, close the file, open it again for output this time (truncating it) and write it back out. This is easiest, but I don't think it's the approach you're looking for.
The other choice is to manipulate the file in place. That's trickier, but not that hard. You need to read about random access I/O (input/output). If you google for c++ random access file you'll get some good hits, but I'll show you a little bit.
// Open the file.
std::ifstream file{"file.dat"};
// Jump to a particular location in the file. Beginning is 0.
file.seekg(128);
// Read 4 bytes
char bytes[4];
file.read(bytes, 4);
// Manipulate it (more below)
int number = bytesToInt(bytes);
++number;
intToBytes(number, bytes);
// Seek again
file.seekg(128);
file.write(bytes, 4);
So the only remaining trick is that you have to convert the bytes to a number and then back into bytes. Due to endianness, it's not safe to read directly into the number. You also need to know the endianness of the data in the file. That's a separate topic you can look up if you're not already familiar with it.
(Specifically, you need to implement those two methods after verifying how the data is stored in your file.)
There may be other ways to do this, but the key to this method is the random access file.
I need to load a txt file with 5 millions data (i.e. strings, just one word with 9 characters per word separated by new line.) into QVector as fast as possible. The code is now working just fine however, if the user hits upload, the application takes 3-5 seconds to load this data for further manipulation. I need to decrease the time of loading this data. What is the right approach to handle this issue? I'm Ok with Qt/STL/Boost. I prefer Qt though. The code that I'm using for this task is the one suggested in Qt documentation which is
QFile file("in.txt");
if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
return;
QTextStream in(&file);
while (!in.atEnd()) {
QString line = in.readLine();
process_line(line);
}
Try this:
I tested it and read file in 2.1 seconds.
I reserve vector before reading and use QElapsedTimer to get reading time.
void MainWindow::readDataText()
{
QString filePath = "F:\\Qt\\Big_File\\Big_File\\data.txt";
QVector<qint64> *vector = new QVector<qint64>;
vector->reserve(5000000);
QElapsedTimer timer;
QFile readFile(filePath);
if(!readFile.open(QFile::ReadOnly | QFile::Text))
{
// Can't Open File.
}
else
{
QByteArray data;
timer.start();
for (int var = 0; var < 5000000; ++var)
{
vector->insert(var, (readFile.readLine()).toInt());
}
qint64 time = timer.elapsed();
ui->txtReadTimeText->setText(QString::number(time));
}
readFile.close();
}
Also it will better if your file being binary.
Another solution is to use readAll() function and read file in 116 miliseconds, and process(split by '\n') data later like this:
void MainWindow::readDataText()
{
QString filePath = "D:\\ProjectTest\\ProjectTest\\data.txt";
QByteArray data;
data.reserve(5000000);
QElapsedTimer timer;
QFile readFile(filePath);
if(!readFile.open(QFile::ReadOnly | QFile::Text))
{
// Can't Open File.
}
else
{
timer.start();
data = readFile.readAll();
qint64 time = timer.elapsed();
ui->txtReadTimeText->setText(QString::number(time));
}
readFile.close();
}
Your example code actually implicitly does decoding. It reads 8-bit encoded text from the file, and converts it to QString, which internally uses 16 bit Unicode encoding.
You will probably gain a big speedup, if instead of using QTextStream, you use just plain QFile directly, and read form it using this readLine method, which return QByteArray, in other words "raw" file contents. The purpose of doing it this way is to avoid creating QString objects for entire file contents.
If you have 5 million lines, then you will also get a significant memory footprint savings, if you store them in memory in QByteArray, instead of QString. Convert to QString only when you are actually going to display the text in the GUI.
Note: Be aware of text encoding! Any text in any file is always encoded, even if especially English-speakers might not realize it. The most straightforward encoding is 7-bit ASCII, a lot of pure English text is actually this, and almost every encoding including UTF-8 is actually superset of 7-bit ASCII, so 7-bit ASCII file can be loaded using almost any encoding. But for multilingual text, you need to know what encoding the file uses, or you will get the accented and other special characters, like ÄÅÁÀÃ, wrong. UTF8 is the only encoding which can store "everything", other encodings such as Latin1 are designed for specific language families.
Note 2: QByteArray actually corresponds to std::string for most purposes. QString is more like std::wstring. Not saying these are identical 1:1 matches, but it helps to think of them as similar.
What I am trying to achieve is this:
Let's say I have a class Score. This class has an int variable and a char* variable.
Now when I have an object Score score, I would like to be able to save the value of those variables (I guess to a file). So now this file has an int variable and a char* variable that I can then access later to create a new Score object.
So I create Score score(10, "Bert");. I either do something like score.SaveScore(); or the score gets saved when the game is over or the program exits, it doesn't matter.
Basically I am looking for the equivalent/correct way of doing this:
score.SaveScore(FILE file)
{
file.var1 = score.score;
file.var2 = score.name;
}
I realize this is probably very stupid and not done this way whatsoever! This is just me trying to explain what I am trying to achieve in the simplest way possible.
Anyway, when I run the program again, that original Score score(10, "Bert") does not exist any more. But I would like to be able to access the saved score(from file or wherever it may be) and create another Score object.
So it may look something like:
LoadScore(FILE file)
{
Score newScore(file.var1, file.var2);
}
Again, just trying to show what I am trying to achieve.
The reason why I want to be able to access the variables again is to eventually have a Scoreboard, the Scoreboard would load a bunch of scores from the file.
Then when a new score is created, it is added to the scoreboard, compared to the other scores currently in the scoreboard and inserted in the right position (like a score of 6 would go in between 9 and 4).
I feel like this was a bit long winded but I was trying to really explain myself well! Which I hope I did!
Anyway, I am not looking for someone to tell me how to do all of that.
All I am after is how to do the initial save to a file.
Thank you for any suggestions.
I would use the <fstream> library, like this;
//example values
int x=10;
float y=10.5;
const char* chars = "some random value";
string str(chars); //make string buffer for sizing
str.resize(20); //make sure its fixed size
//open a test.txt file, in the same dir for output
std::ofstream os("test.txt", std::ios::out | std::ios::binary); //make it output binary
//(char*) cast &x, sizeof(type) for values/write to file chars for x and y
os.write((char*)&x, sizeof(int)); //only sizeof(int) starting at &x
os.write((char*)&y, sizeof(float)); //cast as a char pointer
os.write(str.data(), sizeof(char)*str.size()); //write str data
os.close();
//the file test.txt will now have binary data in it
//to read it back in, just ifstream, and put that info in new containers, like this;
int in_x = 0; //new containters set to 0 for debug
float in_y = 0;
char inchar[20]; //buffer to write 20 chars to
ifstream is("test.txt", std::ios::in | std::ios::binary); //read in binary
is.read((char*)&in_x, sizeof(int)); //write to new containers
is.read((char*)&in_y, sizeof(float));
is.read((char*)&inchar, sizeof(char)*20); //write char assuming 20 size
is.close();
//outputting will show the values are correctly read into the new containers
cout << in_x << endl;
cout << in_y << endl;
cout << inchar << endl;
I realize this is probably very stupid and not done this way whatsoever!
The entire software industry was stupid enough to have it done so many times that even a special term was invented for this operation - serialization and nearly all C++ frameworks and libraries have implemented this in a various ways.
Since question is tagged with C++ I would suggest you to look at boost serialization but there are many other implementations.
Do you need that file to be readable by a human? If yes than consider, for example, XML or JSON formats.
You don't need it be readable but want it be as compact as possible? Consider google protobuf
Just start doing it and come with a more specific question(s).
As it was mentioned before, keep strings as std:string objects rather then char*
About writing/reading to/from files in C++ read about fstream
#everyone, I have some problem in reading data form IMU recently.
Below is the data I got from My device, it is ASCII, all are chars,and my data size is [122], which is really big, I need convert them to short, and then float, but I dont know why and how.....
unsigned char data[33];
short x,y,z;
float x_fl,y_fl,z_fl,t_fl;
float bias[3]={0,0,0};//array initialization
unsigned char sum_data=0;
int batch=0;
if ( !PurgeComm(file,PURGE_RXCLEAR ))
cout << "Clearing RX Buffer Error" << endl;//this if two sentence aim to clean the buffer
//---------------- read data from IMU ----------------------
do { ReadFile(file,&data_check,1,&read,NULL);
//if ((data_check==0x026))
{ ReadFile(file,&data,33,&read,NULL); }
/// Wx Values
{
x=(data[8]<<8)+data[9];
x_fl=(float)6.8664e-3*x;
bias[0]+=(float)x_fl;
}
/// Wy Values
{
y=(data[10]<<8)+data[11];
y_fl=(float)6.8664e-3*y;
bias[1]+=(float)y_fl;
}
/// Wz Values
{
z=(data[12]<<8)+data[13];
z_fl=(float)6.8664e-3*z;
bias[2]+=(float)z_fl;
}
batch++;
}while(batch<NUM_BATCH_BIAS);
$VNYMR,+049.320,-017.922,-024.946,+00.2829,-00.2734,+00.2735,-02.961,+03.858,-08.325,-00.001267,+00.000213,-00.001214*64
$VNYMR,+049.322,-017.922,-024.948,+00.2829,-00.2714,+00.2735,-02.958,+03.870,-08.323,+00.004923,-00.000783,+00.000290*65
$VNYMR,+049.321,-017.922,-024.949,+00.2821,-00.2655,+00.2724,-02.984,+03.883,-08.321,+00.000648,-00.000391,-00.000485*61
$VNYMR,+049.320,-017.922,-024.947,+00.2830,-00.2665,+00.2756,-02.983,+03.874,-08.347,-00.003416,+00.000437,+00.000252*6C
$VNYMR,+049.323,-017.921,-024.947,+00.2837,-00.2773,+00.2714,-02.955,+03.880,-08.326,+00.002570,-00.001066,+00.000690*67
$VNYMR,+049.325,-017.922,-024.948,+00.2847,-00.2715,+00.2692,-02.944,+03.875,-08.344,-00.002550,+00.000638,+00.000022*6A
$VNYMR,+049.326,-017.921,-024.945,+00.2848,-00.2666,+00.2713,-02.959,+03.876,-08.309,+00.002084,+00.000449,+00.000667*6A
all I want to do is:
extract last 6 numbers separated by commas, btw, I don't need the last 3 chars(like *66).
Save the extracted data to 6 .dat files.
What is the best way to do this?
Since I got this raw data from IMU, and I need the last 6 data, which are accelerations(x,y,z) and gyros(x,y,z).
If someone could tell me how to set a counter to the end of each data stream, that will be perfect, because I need the time stamp of IMU also.
Last word is I am doing data acquisition under windows, c++.
Hope someone could help me, I am freaking out because of so much things to do and that's really annoying!!
There's a whole family of scanf functions (fscanf, sscanf and some "secure" ones).
Assuming you have read a line into a string:-
sscanf( s, "VNYMR,%*f,%*f,%*f,%*f,%*f,%*f,%f,%f,%f,%f,%f,%f", &accX, &accY, &accZ, &gyroX, &gyroY, &gyroZ )
And assuming I have counted correctly! This will verify that the literal $VNYMR is there, followed by about five floats that you don't assign and finally the six that you care about. &accaX, etc are the addresses of your floats. Test the result - the number of assignments made..
I am currently working on a project that involves serial communication between a Arduino and a laptop. I know the Arduino is indeed sending the data that I need, see this picture: http://s1.postimg.org/w5wisaetr/Help.png
Now on the other end my laptop is connected to the Arduino and running a program that I made using QT Creator. However, when reading data from the serial Port I can't get the program to display this information.
I connected my readData() function to be executed when data is received like this:
connect(m_serialPort, SIGNAL(readyRead()), m_dataGathering, SLOT(newData()));
This works and the newData() function is called whenever something in transmitted from the Arduino. However the function newData() does not display the data that I need.
newData():
void DataGathering::newData()
{
QByteArray rMsg = m_serial->readAll();
qDebug() << rMsg.constData();
}
This only sends empty message to the display. Like this: http://s2.postimg.org/dkcyip2u1/empty.png
The following code however works:
void DataGathering::newData()
{
QByteArray rMsg("\nTest...");// = m_serial->readAll();
qDebug() << rMsg.constData();
}
This code display the message like it should.
However, another difference in the output display is that when the working code is executed my console also displays a lot of Framing errors, I assumed this is because the baudrate of the unwanted characters differs from the data that I need.
That is why i started questioning the readAll() function.
It is also obvious that the Arduino is not only sending the data that I need but also some unwanted characters (see image in first link), but I don't see this as a problem since I will filter this out later.
All help is very much appreciated.
Update: I found out that the readAll() function is returning QByteArrays with size() equals to 0.
Looks like the serial port QIODevice does not implement bytesAvailable, if it returns 0. This may also be why readAll() fails, depending on how it is implemented. But at least readAll() has the problem of not being able to report error.
Try using read method instead for better diagnostics, like this (untested):
void DataGathering::newData()
{
QByteArray rMsg;
for(;;) {
char buf[256]; // read data in this size chunks
qint64 len = m_serial->read(buf, sizeof buf);
if (len <= 0) {
if (len < 0) {
qDebug() << "newData() read error" << m_serial->errorString();
}
break; // for(;;)
}
rMsg.append(buf, len);
}
qDebug() << "newData() got byte array" << rMsg.size() << ":" << rMsg;
}
It may not solve your problem, but with luck it will give you error message.