How to improve MS Access INSERT performance - c++

I have a C++ program that insert about a million of records into MS Access DB using OLEDBConnection. To do that, I ran the INSERT INTO query a millions time in order to get the records inserted which take quite a long time.
The data is generated in the program in form of array, will that be any other way that i can load the data into database in one single step to improve the performance?
Thanks!
Loop i use to insert the records currently
for (int i = 0; i < populationSize; i++){
insertSQL = "INSERT INTO [" + pTableName + "] (" + columnsName + ") VALUES (" + columnsValue[i] + ");";`
outputDBConn->runSQLEdit(insertSQL);
}
Method that run the SQL query
void DBConnector::runSQLEdit(String^ query){
SQLCMD = gcnew OleDbCommand( query, dbConnection );
SQLCMD->CommandTimeout = 30;
dbConnection->Open();
SQLCMD->ExecuteNonQuery();
dbConnection->Close();
}

It seems very inefficient to open/close the connection for each insert statement.
The standard approach goes something like:
Open connection.
Start transaction, if supported. (This is often very important for databases with transactions.)
Insert. Repeat this step as needed.
Commit transaction, if supported.
Close connection.
Update: The following does not apply to MS Access. Access does not support inserting multiple rows from a literal. It only supports inserting multiple rows from an existing data source. (Although here is a "workabout" that might work. In any case, the most important thing is likely limiting the number of transactions.)
One more thing that can be done is to build a single insert command that adds multiple records at once. This can be done with either multiple statements or a multi-record insert (if supported). It may or may not be significantly faster than just the above (depends upon other factors like network latency and database engine) and will likely need to be adapted to fit within the restrictions of the database (e.g. might only be feasible for a few hundred records at once). This should only be considered after proper connection/transaction usage as described above.
I wouldn't be surprised if there we already-made "bulk insert" libraries/modules floating about... and I don't use MS Access so I can only hope that the above suggestions were helpful :-)
Happy coding.

Don't do ONE insertion per command.
Change your code to something like, this:
string strSQLCommand;
for (int i = 0; i < populationSize; i++){
strSQLCommand += "INSERT INTO [" + pTableName + "] (" + columnsName + ") VALUES (" + columnsValue[i] + ");";`
}
outputDBConn->runSQLEdit(strSQLCommand );
I'm not sure what's the max buffer size of the command, so do some checks and then get the best value to do some "breaks" at every X inserts.

Related

Parallel Writes to NFS-backed File

UPDATE: I had each node write to a separate file, and when the separate files were concatenated together the result was correct. I also updated the code to attempt a channel flush and file sync after each write of a single record, but there are still issues between nodes 0 and 1, now. If I make Node 0 sleep for a few seconds before it starts its iteration of the coforall loop, the records come out correct. If not, the last few hundred bytes of Node 0's records seem to be reliably overwritten with NULL bytes, up to the start of Node 1's records. The issues between Node 1 and Node 2, and Node 2 and Node 3, seem to not show up anymore.
Additionally, if I suppress either Node 0 or Node 1 from writing, I see the fully-formed records from the un-suppressed node written correctly to the file. In the case that Node 1 is suppressed, I see 9,997 100B records (or 999,700) correct bytes followed by NULL bytes in the file where Node 1's suppressed records would go. In the case that Node 0 is suppressed, I see exactly 999,700 NULL bytes in the file, after which Node 1's records begin.
Original Post:
I'm trying to troubleshoot an issue with parallel writes from different nodes to a shared NFS-backed file on disk. At the moment, I suspect that something is wrong with the way writes to the disk happen on the NFS server.
I'm working on adapting MPI+C code that uses pwrite to write to coordinated chunks of a file. If I try to have the equivalent locales in Chapel write to the file inside of a coforall loop, I end up with the bits of the file around the node boundaries messed up - usually the final few hundred bytes of each node's data are garbled. However, if I have just one locale iterate through the data on all locales and write it, the data comes out correctly. That is, I use the same data structures to calculate the offsets, but only Locale 0 seeks to that offset and performs the writes.
I've verified that the offsets into the file that each locale runs do not overlap, and I'm using a single channel per task, defined from within the on loc do block, so that tasks don't share a single channel.
Are there known issues with writing to a file from different locales? A lot of the documentation makes it seem like this is known to be safe, but an unsubstantiated guess seems to indicate that there are issues with caching of file contents; when examining the incorrect data, the bits that are incorrect seem to be the original data from the file in that location at the beginning of the program.
I've included the relevant routine below, in case you easily spot something I missed. To make this serial, I convert the coforall loc in Locales and on loc do block into a for j in 0..numLocales-1 loop, and replace here.id with j. Please let me know what else would help get to the bottom of this. Thanks!
proc write_share_of_data(data_filename: string, ref record_blocks) throws {
coforall loc in Locales {
on loc do {
var data_file: file = open(data_filename, iomode.cwr);
var data_writer = data_file.writer(kind=ionative, locking=false);
var line: [1..100] uint(8);
const indices = record_blocks[here.id].D;
var local_record_offset = + reduce record_blocks[0..here.id-1].D.size;
writeln("Loc ", here.id, ": record offset is ", local_record_offset);
var local_bytes_offset = terarec.terarec_width_disk * local_record_offset;
data_writer.seek(start=local_bytes_offset);
for i in indices {
var write_rec: terarec_t = record_blocks[here.id].records[i];
line[1..10] = write_rec.key;
line[11..98] = write_rec.value;
line[99] = 13; // \r
line[100] = 10; // \n
data_writer.write(line);
lines_written += 1;
}
data_file.fsync();
data_writer.close();
data_file.close();
}
}
return;
}
Adding an answer here that solved my particular problem, though it doesn't explain the behavior seen. I ended up changing the outer loop from coforall loc in Locales to for loc in Locales. This isn't too big of an issue since it's all writing to one file anyway - I doubt that multiple locales can actually make much headway in all attempting to write concurrently to a single file on an NFS server. As a result, the change still allows nodes to write the data they have locally to NFS, rather than forcing Node 0 to collect and then write the data on behalf of all locales. This amounts to only adding idle time to the write operation commensurate with the time it takes Locale 0 to start the remote task on other nodes when the previous node has finished writing, which for the application at hand is not a concern.
Have you tried specifying start/end in file.writer instead of using seek? Does that change anything? What about specifying the end offset for the channel.seek call? Does it matter if the file is created and has the appropriate size before you start?
Other than that, I wonder if this issue would appear for both NFS and Lustre. If it appears for both it might well be a Chapel bug. It sounds from your description that the C program was using this pattern, which points to it being a bug. But, have you run C code doing this on your setup? If it being a Chapel bug seems most likely after further investigation, we would appreciate a bug report issue with a reproducer.
I know that NFS does not always do what one would like, in terms of data consistency. It's my understanding that it has "close to open" semantics but it's unclear to me what that means in the context of opening a file and writing to a particular region within it, in parallel from different locales.
From Why NFS Sucks by Olaf Kirch:
An NFS client is permitted to cache changes locally and send them to
the server whenever it sees fit. This sort of lazy write-back greatly
helps write performance, but the flip side is that everyone else will
be blissfully unaware of these change before they hit the server. To
make things just a little harder, there is also no requirement for a
client to transmit its cached write in any particular fashion, so
dirty pages can (and often will be) written out in random order.
I read two implications from this paragraph that are relevant to your situation here:
The writes you do on different locales can be observed by the NFS server in an arbitrary order. (However as I understand it, the data should be sent to the server by the time your fsync call returns).
These writes are done at an OS page granularity (usually 4k). (Note that this is more a hypothesis I am making than it is a fact. It should be tested or further investigated).
It would be interesting to check if 2. is a plausible explanation for the behavior you are seeing. For example, you could explore having each locale operate on a multiple of 4096 records (or potentially try writing records of 4096 bytes each) and see if that changes the behavior. If 2 is indeed the explanation, it should be possible to create a C program that demonstrates the behavior as well.

MySQL statement is exiting my for loop without finishing the iteration C++

I am working on a project, and I need to push a list to my database. I want to iterate through the said list and execute a statement that adds said list to my table. Running the exact script within MySQL works perfectly fine. However, when I try to iterate through the list and run the script, it does it once before exiting my for loop. It won't even run any code past the executeQuery(). My code is as follows:
for (int i = 0; i < addList.size(); i++) {
stmt = con->createStatement();
stmt->executeQuery("INSERT INTO comboboxtable (combobox, user, text) VALUES (" + std::to_string(id) + ", \'" + username.toStdString() + "\', \'" + addList[i].toStdString() + "\')");
}
I have a driver and connection already established, and everywhere else in my code it seems to execute queries just fine. I also have this code in a try/catch set for catching SQL exceptions, but it isn't throwing anything. I am using the MySQL C++ connector to do this. The size of the loop is fine and everything should be working on the iteration side.
I seem to have figured it out after a bit of digging on the website and found that I needed to use execute() not executeQuery().

ORACLE bulk insert based on SQL

Ich have to shuffle a lot of data data from an application into an ORACLE 11g database. To execute SQL from within C++ I use to employ the following scheme, using bare SQL together with the Poco::Data framework and an ODBC connection to my database:
// get session from ODBC session pool
Session session = moc::SessionPoolMOC::get();
// prepare a statement as QString
QString q = ("INSERT INTO MOC_DATA_CONCENTRATOR"
"("
"EQUIPMENT_INSTANCE_ID, "
"STATION_NAME, "
"DATA_SRC, "
"ORG_ID)"
"values (%1, '%2', '%3', %4);"
);
/*
there would be also prepared statements, but I think that is not the
topic now...
*/
// set parameters within string from passed object 'info'
q = q.arg(info.equipment_instance_id); /* 1 */
q = q.arg(info.station_name.toUtf8().constData()); /* 2 */
q = q.arg(info.data_src.toUtf8().constData()); /* 3 */
q = q.arg(info.org_id); /* 4 */
// prepare statement
Statement query(session);
query << q.toUtf8().constData();
try
{
// execute query
query.execute();
}
catch (...)
{
...
}
I'm aware, that this is a very low level approach, but it worked really fine for all situations I encountered so far...
The problem is, that I have now a lot of data (about 400,000 records) to fill into one table. Data is available as different C++ objects, stored in a Qt-List. My naive approach is to call this code sequence for each object to insert. This works fine, but turns out to be quite slow (taking about 15 minutes or so). I asked around, and was told to better use a so called "bulk insert".
It seems, however, that this is more related to PLSQL - at least I have no Idea how to use that from the same context or in a similar way as shown in my example. I'm neither ORACLE expert nor administrator and usage of the database is only a side problem within my project. What would you recommend?
Many thanks,
Michael

ADO Connection to OLE DB is way too slow

I'm using ADO Connection & Recordset Objects to access a Sybase ASE database (OLE DB Provider)..
For example, simply executing a SQL statement looks something like this:
(inserting 10000 rows of data)
_ConnectionPtr ConnPtr;
ConnPtr.CreateInstance("ADODB.Connection");
ConnPtr->Open(....my Connection String, UserID, and Password....);
for (int i=0; i<10000; i++)
ConnPtr->Execute("INSERT INTO my_table VALUES (1, 2, 3)");
OR (alternative option):
_RecordSet RecPtr; RecPtr.CreateInstance("ADODB.Recordset");
MyObject obj;
// Construct & Bind obj..
...
for (int i=0; i<10000; i++)
RecPtr->AddNew(&obj);
Both approach works fine and produces the expected result.. The only problem is that they are both extremely slow.. Inserting 10000 rows of data using raw sql statements only takes about 3~5 Seconds. On the other hand, accomplishing the same task using ADO objects takes 40-50 Seconds!!!
So here are some of my questions:
Is this a normal result? I mean it's obvious that direct sql execution is always faster than using something like ADO,, but is the performance difference usually this much different??
Can the speed bottleneck be attributed mostly to ADO? Or does the problem has to do more with Database (Sybase)..?
Is there any other way to access OLE DB in C++,, instead of using ADO (Faster alternative)??
Any insights by people who have alot of experience with database please
You should consider using the Prepared property so that the SQL query only gets compiled once. This is slow for a command's first execution, but, you will get improved performance for subsequent executions:
_ConnectionPtr ConnPtr;
ConnPtr.CreateInstance("ADODB.Connection");
ConnPtr->Open(....my Connection String, UserID, and Password....);
_CommandPtr CmdPtr;
CmdPtr.CreateInstance("ADODB.Command");
CmdPtr->ActiveConnection = ConnPtr;
CmdPtr->CommandText = "INSERT INTO my_table VALUES (1, 2, 3)";
CmdPtr->PutPrepared(true);
for (int i=0; i<10000; i++)
CommandPtr->Execute(NULL, NULL, adCmdText);
References:
MSDN - Prepared Property (ADO): http://msdn.microsoft.com/en-us/library/windows/desktop/ms675106(v=vs.85).aspx
MSDN - Prepared Property Example (VC++): http://msdn.microsoft.com/en-us/library/windows/desktop/ms681552(v=vs.85).aspx

Callgrind: Profile a specific part of my code

I'm trying to profile (with Callgrind) a specific part of my code by removing noise and computation that I don't care about.
Here is an example of what I want to do:
for (int i=0; i<maxSample; ++i) {
//Prepare data to be processed...
//Method to be profiled with these data
//Post operation on the data
}
My use-case is a regression test, I want to make sure that the method in question is still fast enough (something like less than 10% extra instructions since the last implementation).
This is why I'd like to have the cleaner output form Callgrind.
(I need a for loop in order to have a significant amount of data processed in order to have a good estimation of the behavior of the method I want to profile)
My first try was to change the code to:
for (int i=0; i<maxSample; ++i) {
//Prepare data to be processed...
CALLGRIND_START_INSTRUMENTATION;
//Method to be profiled with these data
CALLGRIND_STOP_INSTRUMENTATION;
//Post operation on the data
}
CALLGRIND_DUMP_STATS;
Adding the Callgrind macros to control the instrumentation. I also added the --instr-atstart=no options to be sure that I profile only the part of the code I want...
Unfortunately with this configuration when I start to launch my executable with callgrind, it never ends... It is not a question of slowness, because a full instrumentation run last less than one minute.
I also tried
for (int i=0; i<maxSample; ++i) {
//Prepare data to be processed...
CALLGRIND_TOGGLE_COLLECT;
//Method to be profiled with these data
CALLGRIND_TOGGLE_COLLECT;
//Post operation on the data
}
CALLGRIND_DUMP_STATS;
(or the --toggle-collect="myMethod" option)
But Callgrind returned me a log without any call (KCachegrind is white as snow :( and says zero instructions...)
Did I use the macros/options correctly? Any idea of what I need to change in order to get the expected result?
I finally managed to solve this issue... This was a config issue:
I kept the code
for (int i=0; i<maxSample; ++i) {
//Prepare data to be processed...
CALLGRIND_TOGGLE_COLLECT;
//Method to be profiled with these data
CALLGRIND_TOGGLE_COLLECT;
//Post operation on the data
}
CALLGRIND_DUMP_STATS;
But ran the callgrind with --collect-atstart=no (and without the --instr-atstart=no!!!) and it worked perfectly, in a reasonable time (~1min).
The issue with START/STOP instrumentation was that callgrind dumps a file (callgrind.out.#number) at each iteration (each STOP) thus it was really really slow... (after 5min I had only 5000 runs for a 300 000 iterations benchmark... unsuitable for a regression test).
The toggle-collect option is very picky in how you specify the method to use as trigger. You actually need to specify its argument list as well, and even the whitespace needs to match! Use the method name exactly as it appears in the callgrind output. For instance, I am using this invokation:
$ valgrind
--tool=callgrind
--collect-atstart=no
"--toggle-collect=ctrl_simulate(float, int)"
./swaag
Please observe:
The double quotes around the option.
The argument list including parentheses.
The whitespace after the comma character.