I am building a small application. in this i have saved some data in a txt file. i need to edit a value in particular row and column. i wrote a code to go to a particular line and fetch the values but i have tried almost everything to got a particular column and edit that value.
1000 400 120 110 800 110 150 500 0 1000
1000 400 90 150 800 120 150 600 0 1000
1000 400 80 60 **800** 132 150 700 0 1000
1000 400 120 60 800 123 150 200 0 1000
1000 400 111 80 800 143 150 700 0 1000
1000 400 30 90 800 155 150 500 0 1000
for example i have edit the highlighted value , which is the best way to do. i cannot paste my whole code as it is very long.
this is the one where i can go to a particular line
fstream& GotoLine(int num) {
infile.seekg(std::ios::beg);
for (int i = 0; i < num - 1; ++i) {
infile.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
}
return infile;
}
i would appreciate any help over this.
Assuming that your file is formatted as fixed column size, my advice would be:
prepare an array or a vector of struct { size_t pos; size_t width} to define the fields
open the file in ios::in | ios::out mode
read the file one line at a time with your GotoLine function up to the line you want to process
note the index in the fstream with tellg
read the interesting line in a char array with a size greater than the line - here I would use 64 to be large enough - with istream::getline
as you have read the line as a plain array, you can rewrite the characters of one single field
go back to beginning of line with seekp
write the line back to the file
This is not a general method of editing a text file. It only works here because as you have fields of fixed size the edited line has exactly same size than original one, so it can be re-written in place. But never use it in a general case.
Related
Consider the following code snippet of this class template...
template<class T>
class FileTemplate {
private:
std::vector<T> vals_;
std::string filenameAndPath_;
public:
inline FileTemplate( const std::string& filenameAndPath, const T& multiplier ) :
filenameAndPath_( filenameAndPath ) {
std::fstream file;
if ( !filenameAndPath_.empty() ) {
file.open( filenameAndPath_ );
T val = 0;
while ( file >> val ) {
vals_.push_back( val );
}
file.close();
for ( unsigned i = 0; i < vals_.size(); i++ ) {
vals_[i] *= multiplier;
}
file.open( filenameAndPath_ );
for ( unsigned i = 0; i < vals_.size(); i++ ) {
file << vals_[i] << " ";
}
file.close();
}
}
inline std::vector<T> getValues() const {
return vals_;
}
};
When used in main as such with the lower section commented out with the following pre-populated text file:
values.txt
1 2 3 4 5 6 7 8 9
int main() {
std::string filenameAndPath( "_build/values.txt" );
std::fstream file;
FileTemplate<unsigned> ft( filenameAndPath, 5 );
std::vector<unsigned> results = ft.getValues();
for ( auto r : results ) {
std::cout << r << " ";
}
std::cout << std::endl;
/*
FileTemplate<float> ft2( filenameAndPath, 2.5f );
std::vector<float> results2 = ft2.getValues();
for ( auto r : results2 ) {
std::cout << r << " ";
}
std::cout << std::endl;
*/
std::cout << "\nPress any key and enter to quit." << std::endl;
char q;
std::cin >> q;
return 0;
}
and I run this code through the debugger sure enough both the output to the screen and file are changed to
values.txt - overwritten are -
5 10 15 20 25 30 35 40 45
then lets say I don't change any code just stop the debugging or running of the application, and let's say I run this again 2 more times, the outputs respectively are:
values.txt - iterations 2 & 3
25 50 75 100 125 150 175 200 225 250
125 250 375 500 625 750 875 1000 1125 1250
Okay good so far; now lets reset our values in the text file back to default and lets uncomment the 2nd instantiation of this class template for the float with a multiplier value of 2.5f and then run this 3 times.
values.txt - reset to default
1 2 3 4 5 6 7 8 9
-iterations 1,2 & 3 with both unsigned & float the multipliers are <5,2.5> respectively. 5 for the unsigned and 2.5 for the float
- Iteration 1
cout:
5 10 15 20 25 30 35 40 45
12.5 25 37.5 50 62.5 75 87.5 100 112.5
values.txt:
12.5 25 37.5 50 62.5 75 87.5 100 112.5
- Iteration 2
cout:
60
150 12.5 62.5 93.75 125 156.25 187.5 218.75 250 281.25
values.txt:
150 12.5 62.5 93.75 125 156.25 187.5 218.75 250 281.25
- Iteration 3
cout:
750 60
1875 150 12.5 156.25 234.375 312.5 390.625 468.75 546.875 625 703.125
values.txt:
1875 150 12.5 156.25 234.375 312.5 390.625 468.75 546.875 625 703.125
A couple of questions come to mind: it is two fold regarding the same behavior of this program.
The first and primary question is: Are the file read and write calls being done at compile time considering this is a class template and the constructor is inline?
After running the debugger a couple of times; why is the output incrementing the number of values in the file? I started off with 9, but after an iteration or so there are 10, then 11.
This part just for fun if you want to answer:
The third and final question yes is opinion based but merely for educational purposes for I would like to see what the community thinks about this: What are the pros & cons to this type of programming? What are the potentials and the limits? Are their any practical real world applications & production benefits to this?
In terms of the other issues. The main issue is that you are not truncating the file when you do the second file.open statement, you need :
file.open( filenameAndPath_, std::fstream::trunc|std::fstream::out );
What is happening, is that, when you are reading unsigned int from a file containing floating points, it is only reading the first number (e.g. 12.5) up to the decimal place and then stopping (e.g. reading only 12)
, because there is no other text on the line that looks like an unsigned int. This means it only reads the number 12 and then multiplies it by 5 to get the 60, and writes it to the file.
Unfortunately because you don't truncate the file when writing the 60, it leaves the original text at the end which is interpreted as additional numbers in the next read loop. Hence, 12.5 appears in the file as 60 5
stream buffers
Extracts as many characters as possible from the stream and inserts them into the output sequence controlled by the stream buffer object pointed by sb (if any), until either the input sequence is exhausted or the function fails to insert into the object pointed by sb.
(http://www.cplusplus.com/reference/istream/istream/operator%3E%3E/)
I need to read a batch of text files of up to 20mb in size, fast.
The text file comes in the format. The numbers need to be in double format as some other file may have 3 decimal place precision:
0 0 29 175 175 175 175 174
0 1 29 175 175 175 175 174
0 2 29 28 175 175 175 174
0 3 29 28 175 175 175 174
0 4 29 29 175 175 175 174
.
.
.
I would like to store the last six numbers of each line into a single 1D structure like this such that it skips the first two columns. It basically transposes each column and horizontally concatenates each transposed column:
29 29 29 29 29 175 175 28 28 29 175 175 175 175 175...
Here is my class attempting this that is too slow for my purposes.
void MyClass::GetFromFile(std::string filename, int headerLinestoSkip, int ColumnstoSkip, int numberOfColumnsIneed)
{
std::ifstream file(filename);
std::string file_line;
double temp;
std::vector<std::vector<double>> temp_vector(numberOfColumnsIneed);
if(file.is_open())
{
SkipLines(file, headerLinestoSkip);
while(getline(file, file_line, '\n'))
{
std::istringstream ss(file_line);
for(int i=0; i<ColumnstoSkip; i++)
{
ss >> temp;
}
for(int i=0; i<numberOfColumnsIneed; i++)
{
ss >> temp;
temp_vector[i].push_back(temp);
}
}
for(int i=0; i<numberOfColumnsIneed; i++)
{
this->ClassMemberVector.insert(this->ClassMemberVector.end(), temp_vector[i].begin(), temp_vector[i].end());
}
}
I have read that memory mapping the file may be helpful but my attempts to getting it into the 1D structure I need has not been successful. An example from someone would be very much appreciated!
With 20mb and short lines as you show, that's approx 500 000 lines. Knowing this, there are several factors that could slow down your code:
I/O : at the current hardware and OS performance, I can't imagine that this plays a role here;
parsing/conversion. You read each line, build a string stream out of it, to then extract the numbers. This could be an overhead, especially on some C++ implementations where stream extraction is slower than the old sscanf(). I may be wrong but again I'm not sure that this overhead would be so huge.
the memory allocation for your vectors. This is definitely the first place to look for. A vector has a size and a capacity. Each time you add an item above capacity, the vector needs to be reallocated, which could require to move and move again all its content.
I'd strongly advise you to execute your code with a profiler to identify the bottleneck. Manual timing will be difficult here because your loop contains all potential problems, but each iteration is certainly to quick for std::chrono to measure the different loop parts with sufficient accuracy.
If you can't use a profiler, I'd suggest to compute a rough estimation of the number of lines using the file size, and take half of it. Pre-reserve then the corresponding capacity in each temp_vector[i]. If you observe a good progress you'll be the right track and could then fine tune this approach. If not, edit your answer with your new findings and post a comment to this answer.
I am having trouble with split function on QT. I have a text file and the very first line is like this:
10,100;100,100;100,100;100,200;100,200;300,200;300,200;300,350;300,350;250,350;250,350;250,300;250,300;200,300;200,300;200,400;200,400;10,400;10,400;10,100
Here every two numbers represents a point with x an y values and semicolons divides the points. After every semicolon there is a new point.
What I want to do is split these numbers element by element like;
Element 0: 10
Element 1: 100
Element 2: 100
Element 3: 100
...
I have managed to do this but there wasn't any semicolons or colons in the text file.
First line of the text file:
10 100 100 100 100 200 300 200 300 350 250 350 250 300 200 300 200 400 10 400 10 100
This is how i did it:
void Dialog::readFile()
{
QFile file("/Path/of/the/text/file/Data.txt");
if(!file.open(QIODevice::ReadOnly)) {
QMessageBox::information(0, "info", file.errorString());
}
QString line = file.readLine();
qDebug() << "line:" << line << "\n";
QStringList list = line.split(" ");
double element;
for(int i = 0; i < list.size(); i++) {
element = list.at(i).toDouble();
points_.append(element);
qDebug() << "element:" << element << "\n";
}
}
Output of this code is:
line: "10 100 100 100 100 200 300 200 300 350 250 350 250 300 200 300 200 400 10 400 10 100\r"
element: 10
element: 100
element: 100
element: 100
element: 100
element: 200
element: 300
element: 200
....
I want to do this exactly the same way. Is there any suggestion?
PS: I am new on QT so please consider this.
You can use regular expression with the QString::split overload taking QRegExp or the next one taking QRegularExpression:
QStringList list = line.split(QRegExp(",|;")); // ",|;" matches ',' or ';'
QRegExp is called to be deprecated, but it can do the job here.
I have a DataFrame with 80,000 rows. One column 'prod_prom' contains either null values or string representations of numbers, i.e. including ','. I need to convert these to integers. So far I have been doing this:
for row in DF.index:
if pd.notnull(DF.loc[row, 'prod_prom']):
DF.loc[row, 'prod_prom'] = int(''.join([char for char in DF.loc[row, 'prod_prom'] if char != ',']))
But it is extremely slow. Would it be quicker to do this in list comprehension, or with an apply function? What is best practice for this kind of operation?
Thanks
So if I understand right, you have data like the following:
data = """
A,B
100,"5,000"
200,"10,000"
300,"100,000"
400,
500,"2,000"
"""
If that is the case probably the easiest thing is to use the thousands option in read_csv (the type will be float instead of int because of the missing value):
df = pd.read_csv(StringIO(data),header=True,thousands=',')
A B
0 100 5000
1 200 10000
2 300 100000
3 400 NaN
4 500 2000
If that is not an possible you can do something like the following:
print df
A B
0 100 5,000
1 200 10,000
2 300 100,000
3 400 NaN
4 500 2,000
df['B'] = df['B'].str.replace(r',','').astype(float)
print df
A B
0 100 5000
1 200 10000
2 300 100000
3 400 NaN
4 500 200
I changed the type to float because there are no NaN integers in pandas.
Hello Stack Overflow world :3 My name is Chris, I have a slight issue.. So I am going to present the issue in this format..
Part 1
I will present the materials & code snippets I am currently working with that IS working..
Part 2
I will explain in my best ability my desired new way of achieving my goal.
Part 3
So you guys think I am not having you do all the work, I will go ahead and present my attempts at said goal, as well as possibly ways research has dug up that I did not fully understand.
Part 1
mobDB.csv Example:
ID Sprite kName iName LV HP SP EXP JEXP Range1 ATK1 ATK2 DEF MDEF STR AGI VIT INT DEX LUK Range2 Range3 Scale Race Element Mode Speed aDelay aMotion dMotion MEXP ExpPer MVP1id MVP1per MVP2id MVP2per MVP3id MVP3per Drop1id Drop1per Drop2id Drop2per Drop3id Drop3per Drop4id Drop4per Drop5id Drop5per Drop6id Drop6per Drop7id Drop7per Drop8id Drop8per Drop9id Drop9per DropCardid DropCardper
1001 SCORPION Scorpion Scorpion 24 1109 0 287 176 1 80 135 30 0 1 24 24 5 52 5 10 12 0 4 23 12693 200 1564 864 576 0 0 0 0 0 0 0 0 990 70 904 5500 757 57 943 210 7041 100 508 200 625 20 0 0 0 0 4068 1
1002 PORING Poring Poring 1 50 0 2 1 1 7 10 0 5 1 1 1 0 6 30 10 12 1 3 21 131 400 1872 672 480 0 0 0 0 0 0 0 0 909 7000 1202 100 938 400 512 1000 713 1500 512 150 619 20 0 0 0 0 4001 1
1004 HORNET Hornet Hornet 8 169 0 19 15 1 22 27 5 5 6 20 8 10 17 5 10 12 0 4 24 4489 150 1292 792 216 0 0 0 0 0 0 0 0 992 80 939 9000 909 3500 1208 15 511 350 518 150 0 0 0 0 0 0 4019 1
1005 FARMILIAR Familiar Familiar 8 155 0 28 15 1 20 28 0 0 1 12 8 5 28 0 10 12 0 2 27 14469 150 1276 576 384 0 0 0 0 0 0 0 0 913 5500 1105 20 2209 15 601 50 514 100 507 700 645 50 0 0 0 0 4020 1
1007 FABRE Fabre Fabre 2 63 0 3 2 1 8 11 0 0 1 2 4 0 7 5 10 12 0 4 22 385 400 1672 672 480 0 0 0 0 0 0 0 0 914 6500 949 500 1502 80 721 5 511 700 705 1000 1501 200 0 0 0 0 4002 1
1008 PUPA Pupa Pupa 2 427 0 2 4 0 1 2 0 20 1 1 1 0 1 20 10 12 0 4 22 256 1000 1001 1 1 0 0 0 0 0 0 0 0 1010 80 915 5500 938 600 2102 2 935 1000 938 600 1002 200 0 0 0 0 4003 1
1009 CONDOR Condor Condor 5 92 0 6 5 1 11 14 0 0 1 13 5 0 13 10 10 12 1 2 24 4233 150 1148 648 480 0 0 0 0 0 0 0 0 917 9000 1702 150 715 80 1750 5500 517 400 916 2000 582 600 0 0 0 0 4015 1
1010 WILOW Willow Willow 4 95 0 5 4 1 9 12 5 15 1 4 8 30 9 10 10 12 1 3 22 129 200 1672 672 432 0 0 0 0 0 0 0 0 902 9000 1019 100 907 1500 516 700 1068 3500 1067 2000 1066 1000 0 0 0 0 4010 1
1011 CHONCHON Chonchon Chonchon 4 67 0 5 4 1 10 13 10 0 1 10 4 5 12 2 10 12 0 4 24 385 200 1076 576 480 0 0 0 0 0 0 0 0 998 50 935 6500 909 1500 1205 55 601 100 742 5 1002 150 0 0 0 0 4009 1
So this is an example of the Spreadsheet I have.. This is what I wish to be using in my ideal goal. Not what I am using right now.. It was done in MS Excel 2010, using Columns A-BF and Row 1-993
Currently my format for working code, I am using manually implemented Arrays.. For example for the iName I have:
char iName[16][25] = {"Scorpion", "Poring", "Hornet", "Familiar", "null", "null", "null", "null", "null", "null", "null", "null", "null", "null", "null", "null"};
Defined in a header file (bSystem.h) now to apply, lets say their health variable? I have to have another array in the same Header with corresponding order, like so:
int HP[16] = {1109, 50, 169, 155, 95, 95, 118, 118, 142, 142, 167, 167, 193, 193, 220, 220};
The issue is, there is a large amount of data to hard code into the various file I need for Monsters, Items, Spells, Skills, ect.. On the original small scale to get certain system made it was fine.. I have been using various Voids in header files to transfer data from file to file when it's called.. But when I am dealing with 1,000+ Monsters and having to use all these variables.. Manually putting them in is kinda.. Ridiculous? Lol...
Part 2
Now my ideal system for this, is to be able to use the .CSV Files to load the data.. I have hit a decent amount of various issues in this task.. Such as, converting the data pulled from Names to a Char array, actually pulling the data from the CSV file and assigning specific sections to certain arrays... The main idea I have in mind, that I can not seem to get to is this;
I would like to be able to find a way to just read these various variables from the CSV file... So when I call upon the variables like:
cout << name << "(" << health << " health) VS. " << iName[enemy] << "(" << HP[enemy] << " health)";
where [enemy] is, it would be the ID.. the enemy encounter is in another header (lSystem.h) where it basically goes like;
case 0:
enemy = 0;
Where 0 would be the first data in the Arrays involving Monsters.. I hate that it has to be order specific.. I would want to be able to say enemy = 1002; so when the combat systems start it can just pull the variables it needs from the enemy with the ID 1002..
I always hit a few different issues, I can't get it to pull the data from the file to the program.. When I can, I can only get it to store int values to int arrays, I have issues getting it to convert the strings to char arrays.. Then the next issue I am presented with is recalling it and the actual saving part... Which is where part 3 comes in :3
Part 3
I have attempted a few different things so far and have done research on how to achieve this.. What I have came across so far is..
I can write a function to read the data from let's say mobDB, record it into arrays, then output it to a .dat? So when I need to recall variables I can do some from the .dat instead of a modifiable CSV.. I was presented with the same issues as far as reading and converting..
I can go the SQL route, but I have had a ton of issues understanding how to pull the data from the SQL? I have a PowerEdge 2003 Server box in my house which I store data on, it does have NavicatSQL Premium set up, so I guess my main 2 questions about the SQL route is, is it possible to hook right into the SQLServer and as I update the Data Base, when the client runs it would just pull the variables and data from the DB? Or would I be stuck compiling SQL files... When it is an online game, I know I will have to use something to transfer from Server to Client, which is why I am trying to set up this early in dev so I have more to build off of, I am sure I can use SQL servers for that? If anyone has a good grasp on how this works I would very much like to take the SQL route..
Attempts I have made are using like, Boost to Parse the data from the CSV instead of standard libs.. Same issues were presented.. I did read up on converting a string to a char.. But the issue lied in once I pulled the data, I couldn't convert it?..
I've also tried the ADO C++ route.. Dead end there..
All in all I have spent the last week or so on this.. I would very much like to set up the SQL server to actually update the variables... but I am open to any working ideas that presents ease of editing and implementing large amounts of data..
I appreciate any and all help.. If anyone does attempt to help get a working code for this, if it's not too much trouble to add comments to parts you feel you should explain? I don't want someone to just give me a quick fix.. I actually want to learn and understand what I am using. Thank you all very much :)
-Chris
Let's see if I understand your problem correctly: You are writing a game and currently all the stats for your game actors are hardcoded. You already have an Excel spreadsheet with this data and you just want to use this instead of the hardcoded header files, so that you can tweak the stats without waiting for a long recompilation. You are currently storing the stats in your code in a column-store fashion, i.e. one array per attribute. The CSV file stores stuff in a row-wise fashion. Correct so far?
Now my understanding of your problem becomes a little blurry. But let's try. If I understand you correctly, you want to completely remove the arrays from your code and directly access the CSV file when you need the stats for some creature? If yes, then this is already the problem. File I/O is incredibly slow, you need to keep this data in main memory. Just keep the arrays, but instead of manually assigning the values in the headers, you have a load function that reads the CSV file when you start the game and loads its contents into the array. You can keep the rest of your code unchanged.
Example:
void load (std::ifstream &csv)
{
readFirstLineAndCheckThatItIsCorrect (csv);
while (!csv.eof())
{
int id;
std::string spriteName;
csv >> id;
csv >> spriteName >> kName[id] >> iName[id] >> LV[id] >> HP[id] >> SP[id] >> ...
Sprite[id] = getSpriteForName (spriteName);
}
}
Using a database system is completely out of scope here. All you need to do is load some data into some arrays. If you want to be able to change the stats without restarting the program, add some hotkey for reloading the CSV file.
If you plan to write an online game, then you still have a long way ahead of you. Even then, SQL is a very bad idea for exchanging data between server and clients because a) it just introduces way too much overhead and b) it is an open invitation for cheaters and hackers because if clients have direct access to your database, you can no longer validate their inputs. See http://forums.somethingawful.com/showthread.php?noseen=0&pagenumber=258&threadid=2803713 for an actual example.
If you really want this to be an online game, you need to design your own communication protocol. But maybe you should read some books about that first, because it really is a complex issue. For instance, you need to hide the latency from the user by guessing on the client side what the server and the other players will most likely do next, and gracefully correct your guesses if they were wrong, all without the player noticing (Dead Reckoning).
Still, good luck on your game and I hope to play it some day. :-)
IMO, the simplest thing to do would be to first create a struct that holds all the data for a monster. Here's a reduced version because I don't feel like typing all those variables.
struct Mob
{
std::string SPRITE, kName, iName;
int ID, LV, HP, SP, EXP;
};
The loading code for your particular format is then fairly simple:
bool ParseMob(const std::string & str, Mob & m)
{
std::stringstream iss(str);
Mob tmp;
if (iss >> tmp.ID >> tmp.SPRITE >> tmp.kName >> tmp.iName
>> tmp.LV >> tmp.HP >> tmp.SP >> tmp.EXP)
{
m = tmp;
return true;
}
return false;
}
std::vector<Mob> LoadMobs()
{
std::vector<Mob> mobs;
Mob tmp;
std::ifstream fin("mobDB.csv");
for (std::string line; std::getline(fin, line); )
{
if (ParseMob(line,tmp))
mobs.emplace_back(std::move(tmp));
}
return mobs;
}