TL;DR I am trying to take a stream of data and make it write to a .csv file. Everything is worked out except the writing part, which I think is simply due to me not referencing the .csv file correctly. But I'm a newbie to this stuff, and can't figure out how to correctly reference it, so I need help.
Hello, and a big thank you in advance to anyone that can help me out with this! Some advance info, my IDE is Xcode, using C++, and I'm using the Myo armband from Thalmic Labs as a device to collect data. There is a program (link for those interested enough to look at it) that is supposed to stream the EMG, accelerometer, gyroscope, and orientation values into a .csv file. I am so close to getting the app to work, but my lack of programming experience has finally caught up to me, and I am stuck on something rather simple. I know that the app can stream the data, as I have been able to make it print the EMG values in the debugging area. I can also get the app to open a .csv file, using this code:
const char *path= "/Users/username/folder/filename";
std::ofstream file(path);
std::string data("data to write to file");
file << data;
But no data ends up being streamed/printed into that file after I end the program. The only thing that I can think might be causing this is that the print function is not correctly referencing this file pathway. I would assume that to be a straightforward thing, but like I said, I am inexperienced, and do not know exactly how to address this. I am not sure what other information is necessary, so I'll just provide everything that I imagine might be helpful.
This is the function structure that is supposed to open the files: (Note: The app is intended to open the file in the same directory as itself)
void openFiles() {
time_t timestamp = std::time(0);
// Open file for EMG log
if (emgFile.is_open())
{
emgFile.close();
}
std::ostringstream emgFileString;
emgFileString << "emg-" << timestamp << ".csv";
emgFile.open(emgFileString.str(), std::ios::out);
emgFile << "timestamp,emg1,emg2,emg3,emg4,emg5,emg6,emg7,emg8" << std::endl;
This is the helper to print accelerometer and gyroscope data (There doesn't appear to be anything like this to print EMG data, but I know it does, so... Watevs):
void printVector(std::ofstream &path, uint64_t timestamp, const myo::Vector3< float > &vector)
{
path << timestamp
<< ',' << vector.x()
<< ',' << vector.y()
<< ',' << vector.z()
<< std::endl;
}
And this is the function structure that utilizes the helper:
void onAccelerometerData(myo::Myo *myo, uint64_t timestamp, const myo::Vector3< float > &accel)
{
printVector(accelerometerFile, timestamp, accel);
}
I spoke with a staff member at Thalmic Labs (the guy who made the app actually) and he said it sounded like, unless the app was just totally broken, I was potentially just having problems with the permissions on my computer. There are multiple users on this computer, so that may very well be the case, though I certainly hope not, and I'd still like to try and figure it out one more time before throwing in the towel. Again, thanks to anyone who can be of assistance! :)
My imagination is failing me. Have you tried writing to or reading from ostringstream or istringstream objects? That might be informative. Here's a line that's correct:
std::ofstream outputFile( strOutputFilename.c_str(), std::ios::app );
Note that C++ doesn't have any native support for streaming .csv code, though, you may have to do those conversions yourself. :( Things may work better if you replace the "/"'s by (doubled) "//" 's ...
Related
I have been scratching my head around this problem for a while, and I couldn't find any answer through surfing the web either.
The problem is that I call system("csvtojson someFile.csv 1> someOtherFile.json") inside my program to produce a JSON file. After this line I want to open, read, and process the JSON file. Although, I can see that the file is created, but fopen() returns NULL.
I read that system() is synchronized so I think the rest of my program will not get executed until the system call is finished, and so the file will be created.
I suspect the problem is somehow related to redirecting the output stream using "1>"; not sure, though.
Any help or hint will be much appreciated.
Thanks! :)
P.S. I don't want to use a library to convert csv to JSON, and I can't perform the conversion outside the program because there are tons of very large csv files and the only way for me is to convert each to a JSON file inside the program, run my algorithm, and move to the next csv file ( converting it to JSON and saving it in the very same JSON file). So in total I have only one JSON file, being like a buffer for my csv files. Having said that, if anyone has a better design approach that can be implemented quickly, that would be also great.
UPDATE : Actual code that exhibits the problem, copied from the OP's answer:
int main(){
system("csvtojson Test_Trace.csv 1> ~/Traces/Test_Trace.json");
FILE* traceFile = fopen("~/Traces/Test_Trace.json", "r");
if(traceFile == NULL)
perror("Error in Openning the trace file");
else
cout << "Successfull openning of the trace file!" << endl;
return 0;
}
Thank you guys for your answers. I had to be more detailed in my question as the problem seemed to be somewhere that wasn't clear from my question.
I figured out what was the problem, and would like to share it here (not a super interesting finding, but worth mentioning).
I wrote a simple program to find the problem:
int main(){
system("csvtojson Test_Trace.csv 1> ~/Traces/Test_Trace.json");
FILE* traceFile = fopen("~/Traces/Test_Trace.json", "r");
if(traceFile == NULL)
perror("Error in Openning the trace file");
else
cout << "Successfull openning of the trace file!" << endl;
return 0;
}
If you run this program you will get the error message No such file or directory, but if you replace the address string with the absolute location, i.e., /home/USER_ID/Traces/Test_Trace.json, in both system(...) and fopen(...) calls, your code will work fine. Interestingly, myself suspected that this could be the problem and I changed just the one for system(...) but still it wasn't working (though the file was being created in the location that was passed to fopen(...)).
EDIT: Thanks to #Peter's comment, this problem was because system() call takes care of ~, but fopen() does not and need an absolute path. So there is really no need to have both functions been given the absolute path.
Anyhow,
Thanks Again. :)
Perhaps the reason for this is because the system command hasn't finished executing by the time your program continues to the next instructions where it tries to read from the file that hasn't been created yet.
Although, this isn't the best way, putting in a short pause might make the difference, or at least let you know if that is the issue.
Logger for my program. I saw in another program that it’s somehow possible to open and read a file with text editor while the program is still using it. Seems it just opens a copy for me and continue logging in the background. This kind of log system I need too. But if I use fopen() I only can open and read the file with my text editor if the Programm already closed it with fclose(); This way would work but I think its a very bad solution and also very slow... to open and close the file on every log :S
Someone knows how the needed log system is working?
P.S. I'm working in VisualStudio 2013 on Windows 8.1
Sry for my bad English :S
There are 2 different problems.
First is writing of logs. In a Windows system, the buffering will cause the data to be actually written to disk :
if you close the file
when you have a fair quantity of new data (unsure between several ko and several Mo)
if you explicitely flush
Unless if you have a high throughput, I would advise to at least flush (if not close) after each write to avoid loosing logs if program crashes. And it also allows you to read the log file in real time.
Second is reading. Vim for example is known to be able to monitor a file that can be modified by an external process. It will open a popup saying that file has been modified and offer to reload it. I do not know what notepad does in same conditions. But :
it does not have sense unless first problem has gone
it is not very efficient since you will reload whole file each time
IMHO, you'd better write a custom reader that mimics Linux tail -f :
read (and display) until end of file
repeteadly read (with a short sleep after an unsuccessful read) to process newly added data
It all depends on the text editor you are using. Some will notice edit to the file and ask you if you want to reload a fresh version.
If you work on linux, and you'd like to have an idea of what's happening in real time you could do someting like
tail -f <path-to-file>
or if the file doesnt yet exist
watch -n 0,2 "cat <path-to-file> | tail"
which will display the content of the file and refresh it every 0.2 sec
Thx for your fast answers :)
Crazy.. i was working so long with fopen() and found no solution.. also the fflush(pFile) didnt help (I wasnt able to open file.. always error that its already in use by another program). I never tryed the fstream. Seems fstream solved my problem now. I can open my file with msnotepad.exe while the program is still writing to the file :) Here a small test-code:
#include <fstream> #include <iostream> using namespace std;
int main(){
ofstream FILE;
FILE.open("E:\\Log.txt");
for (size_t i = 0; i < 50; i++)
{
FILE << "Hello " << i << endl;
cout << "log" << endl;
_sleep(500);
}
FILE.close();
cout << "finish" << endl;
return 0;}
I'm working on an assignment for my computer science class, its a first year course as I'm a beginner and I am having trouble with a certain part.
A quick explanation of what my assignment does is:
It takes information from a text file and puts it in a vector while the program is running, and you can add names to it or remove names from it, and once you are done you need it to save the information, which means you have to take the information back out of the vector and replace it into the text file.
I haven't learned of a way to take information out of a vector and back into a text-file, I saw that a classmate of mine posted on here a few times but he was pretty much dismissed so he told me to ask the question for myself.
We were given a bit of the coding for our program and honestly I have got no clue on how to make the function take the information back out of the vector and into the text file updated.
What ive included:
#include <iostream>
#include <string>
#include <vector>
#include <fstream>
here is the function in which it would save into, any help would be greatly appreciated.
void quit_program(string strFileName, vector<string>&vecThisStudent)
{
//write the body of the function
//Hint: this is the function you should write the vector’s content back to the file.
cout<<"Thanks for using the program. Program terminated"<<endl;
}
As you can see we were even given the hint on what the function was supposed to do, but anyone I have spoken to from the class hasnt had a clue on how to get it done (or they dont like me enough to tell me)
If the entire program is needed, I can post it. It looks almost identical to my classmate who posted earlier, but that is just because we were given the majority of the code and we just had to complete a few different things, and I've just been stuck here for the last 10 hours or so.
My read_file
int read_file(string strFileName, vector<string>&vecThisStudent, fstream &inFile)
{
string strFirstName
string strLastName;
inFile.open(strFileName.c_str(), ios::in | ios::out | ios::app);
while (inFile >> strFirstName >> strLastName)
{
vecThisStudent.push_back(strFirstName + " " + strLastName);
}
return 0;
}
Split the problem into sub-problems. Keep splitting to a smaller pieces till each piece is manageable.
In your case sub-problems I would be comfortable working with are "C++ performing action at program exit", "C++ container serialize", "C++ file IO".
The first one will give you C: Doing something when the program exits, the second - ostream iterator usage in c++, and finally the third one - Writing in file C++.
As a final step you just need to combine all three back together.
And Steve, do not blame your professor or your destiny. Being a good programmer is as hard as being a good surgeon, as hard and as rewarding, but requires quite a bit of dedication to grow from mediocrity to a sharp Swiss Army Knife. At your first job interview you'll see how much worse questions can be than ones asked in these assignments.
Seeing your lack of C++ knowledge, I would REALLY suggest watching some tutorials about C++. If you don't know what a for-loop is/how to use it, you will have MAJOR problems with future assignments.
Here are some great series of tutorial.
There's no such thing are taking the contents of a file (or vector) and placing it automatically into a vector (or file).
But to read or write data, take a look at this page.
The general idea of reading a file is:
Iterate though the file and read each input one by one.
Place that input into a vector
The general idea of outputting data to a file is:
Iterate though the data (ex: every element of that vector)
Output that data (ex: that element).
By iterating, I mean running though the data (usually by a for-loop):
int write_file(string strFileName, vector<string>&vecThisStudent, fstream &outFile)
{
outFile.open(strFileName.c_str(), ios::in | ios::out | ios::app);
for (int i = 0 ; i < vecThisStudent.size() ; i++) {
//Use this line to output to console
cout << vecThisStudent[i] << " \n";
//Use this line to output to file
outFile << vecThisStudent[i] << "\n";
}
}
Use ofstream
http://www.cplusplus.com/reference/fstream/ofstream/
Open File..
Write data using << (http://www.cplusplus.com/reference/ostream/ostream/operator%3C%3C/)
Close file..
I am not sure what exactly you stuck with..
I have the following code:
ofstream mOutFile.open(logPath, ios_base::app);
string lBuilder;
lBuilder.append("========================================================\n");
lBuilder.append("Date: ");
lBuilder.append(asctime(timeinfo));
lBuilder.append("\n");
lBuilder.append("Log Message:\n");
lBuilder.append(toLog);
lBuilder.append("\n");
lBuilder.append("========================================================\n\n");
int lSize = lBuilder.size();
char* lBuffer = new char[lSize];
int index = 0;
for each (char c in lBuilder)
lBuffer[index++] = c;
mOutFile.write(lBuffer, lSize);
mOutFile.flush();
Unfortunately, until I close the app (I assume that closing the ofstream would work as well) the output does not get written to the text file. I know I could probably close and reopen the stream and everything will "just work" but that seems like a silly and incorrect solution. What am I doing wrong here?
I have also tried the following variations based on other questions I have found here, but these solutions did not work:
mOutputFile << flush;
mOutputFile << endl;
Thanks in advance for any assistance on this.
edit Everything in this code is working visual c++, it builds and works fine except the file is not written to until the stream is closed, even if I force a flush. Also, I switched from using the << operator to the char * and .write () to see if anything behaved differently.
std::ofstream file(logPath, ios_base::app);
file << "========================================================\n"
<< "Date: " << asctime(timeinfo)
<< "\nLog Message:\n" << toLog
<< "\n========================================================\n\n"
<< std::flush;
//if you want to force it write to the file it will also flush when the the file object is destroyed
//file will close itself
This is not only easier to read but it will probably also be faster than your method + it is a more standard appraoch
I ended up just "making it work" by closing and reopening the stream after the write operation.
mOutputFile << "all of my text" << endl;
mOutputFile.close();
mOutputFile.open(mLogPath);
EDIT After trying out forcing the flush on a few other systems, it looks like something just isn't performing correctly on my development machine. Not good news but at least the above solution seems to work when programmatically flushing the ofstream fails. I am not sure of the implications of the above code though, so if anyone wants to chime in if there are implications of closing and reopening the stream like this.
You can perform the following steps to validate some assumptions:
1.) After flush(), the changes to the file should be visible to your application. Open the file as std::fstream instead of std::ofstream. After flushing, reset the file pointer to the beginning and read the contents of the file. Your newly written record should be there. If not, you probably have a memory corruption somewhere in your code.
2.) Open the same file in an std::ifstream after your call to flush(). Then read the contents of the file. Your newly written record should be there. If not, then there's probably another process interfering with your file.
If both works, then you may want to read up on "file locking" and "inter-process syncronization". The OS can (theoretically) take as much time as it wants to make file changes visible to other processes.
I'm in a embedded led measuring system project now. It uses ARM & linux, and has 64M memory and 1G storage. When measuring, it's supposed to write data to a .csv file. I did it this way:
Create/open a file before measurement begins
In the measuring loop, when data is ready, put it into the file, then go to next measuring
When user stop the measurement, the file will be closed
But, when I add this feature, the program keep running several hours, then the machine won't respond to anything ( measuring stopped, UI still display but doesn't respond to any action, etc.). And the csv file is about 15MB.
While without this feature, the machine can work well all day.
I've thought about this, maybe It's because the memory is used up. With such a small memory, is it possible to keep writing a file? Or should I close it every time I finished writing data? (In that case, I will have to open/close the file very frequently, it will cause our system to be slow, what is not glad to see)
Apologize for my poor English, maybe someone can understand it and give me some help.
God is lighting your path, thank you all!
ps: I do believe the file operations itself is correct.
the code like this:
std::ofstream out_put;
out_put.open(filePath, std::ofstream::out | std::ofstream::trunc);
while(!userStoped()){
doSomeMesuring();
for(int itemIndex = 0; itemIndex < itemCount; ++itemIndex){
out_put << ',' << itemName.toStdString() << ','
<< data->mdata.item[itemIndex].mvalue << ','
<< data->mdata.item[itemIndex].judge << std::endl;
}
}
out_put.close();
You write to 'out_put', the ofstream, but never check if the stream is still valid.
You could change it to
while (out_put.good() && (!userStoped())
To prove to yourself that it is the writing to a stream which is causing the problem, comment out all of the measuring code, just write lots of 'x' (or your choice of character!) to the stream to see if you have the same result.