Ich have to shuffle a lot of data data from an application into an ORACLE 11g database. To execute SQL from within C++ I use to employ the following scheme, using bare SQL together with the Poco::Data framework and an ODBC connection to my database:
// get session from ODBC session pool
Session session = moc::SessionPoolMOC::get();
// prepare a statement as QString
QString q = ("INSERT INTO MOC_DATA_CONCENTRATOR"
"("
"EQUIPMENT_INSTANCE_ID, "
"STATION_NAME, "
"DATA_SRC, "
"ORG_ID)"
"values (%1, '%2', '%3', %4);"
);
/*
there would be also prepared statements, but I think that is not the
topic now...
*/
// set parameters within string from passed object 'info'
q = q.arg(info.equipment_instance_id); /* 1 */
q = q.arg(info.station_name.toUtf8().constData()); /* 2 */
q = q.arg(info.data_src.toUtf8().constData()); /* 3 */
q = q.arg(info.org_id); /* 4 */
// prepare statement
Statement query(session);
query << q.toUtf8().constData();
try
{
// execute query
query.execute();
}
catch (...)
{
...
}
I'm aware, that this is a very low level approach, but it worked really fine for all situations I encountered so far...
The problem is, that I have now a lot of data (about 400,000 records) to fill into one table. Data is available as different C++ objects, stored in a Qt-List. My naive approach is to call this code sequence for each object to insert. This works fine, but turns out to be quite slow (taking about 15 minutes or so). I asked around, and was told to better use a so called "bulk insert".
It seems, however, that this is more related to PLSQL - at least I have no Idea how to use that from the same context or in a similar way as shown in my example. I'm neither ORACLE expert nor administrator and usage of the database is only a side problem within my project. What would you recommend?
Many thanks,
Michael
Related
I'm writing a program which updates MySQL tables every few seconds as new data is received. The function has an infinite loop to update constantly, with another loop inside it that iterates through table rows and picks data accordingly.
Unfortunately, no matter what I do, I encounter the most pathetic error:
Can't create more than max_prepared_stmt_count statements
I've tried simply deleting all mysql related values (except con and driver) after each iteration with statements seen in the example code here.
I've tried adding
stmt = NULL;
stmt->close();
Just in case before every value.
I've tried rewriting the program with auto_ptr, which allegedly clean up automatically, based on this answer.
I have even tried closing and reestablishing connection to the MySQL server after each iteration, but even that did not help me.
Please help.
More information on your loop may be required.
But it sounds like your stmt = con->createStatement(); is inside your loop. You only need one connection, before your loop starts.
Then you can execute a statment multiple times in your loop, res = stmt->executeQuery("SELECT 'Hello World!' AS _message");
When your application terminates then you close your stmt, delete stmt;
From mysql.com "The default value is 16,382." mysql.com-sysvar_max_prepared_stmt_count Which it sounds like you only require one.
I'm searching for a way to call an anonymous PLsql block through SOCI. The data transfer takes place through a refcursor that was previously created as a variable in the script:
variable rc refcursor
declare
v_obj_id number(4,0) := 1;
v_obj_def varchar(30);
v_obj_type number := 1;
begin
open :rc for
select v_obj_id, OBJ_DEF_ID
from MY_OBJECT_DEFS
where OBJECT_TYPE = v_obj_type;
end;
I need to read the refcursor from my application to retrieve the data. I tried to execute the above through a soci::statement but it gives me the error: ORA-24333: zero iteration count. The PLsql script works fine when executed in SqlPlus.
How can I make the connection between the statement and the
refcursor rc? Should I use some other SOCI construct (other than statement) for this purpose?
I understand there are two instructions in the above
script; (i. the refcursor creation, ii. the anonymous PLsql block
itself). I'm not sure whether its possible to call multiple
instructions in a single SOCI statement. Can this be confirmed?
Following is the what I tried. The sSQL contains the above PLsql script:
dbConn.open("...");
int iObjId;
std::string iObjDefId;
soci::indicator ind_iObjId = soci::i_ok,
ind_iObjDefId = soci::i_ok;
soci::statement stmt(dbConn);
stmt.alloc();
stmt.prepare(sSQL);
stmt.exchange(soci::into(iObjId, ind_iObjId));
stmt.exchange(soci::into(iObjDefId, ind_iObjDefId));
stmt.define_and_bind();
stmt.execute(false);
while (stmt.fetch())
{
if (soci::i_ok == ind_iObjId)
std::cout << "Obj ID: " << iObjId << std::endl;
if (soci::i_ok == ind_iObjDefId)
std::cout << "Obj Def ID: " << iObjDefId << std::endl;
}
EDIT: I'm using Oracle 11g
The statement variable rc refcursor is neither SQL nor PL/SQL but part of Oracle's SQL*Plus command-line utility and compatible third party products. I don't know C++ but presumably you would need to define a ref cursor object in the host program.
If this is not feasible, and you are on Oracle 12.1 or later, it's possible that you could use an implicit result set construction, along the lines of
declare
rc sys_refcursor;
begin
open rc for select * from dual;
dbms_sql.return_result(rc);
end;
as discussed in Is it possible to output a SELECT state from a PL/SQL block?
I am using the SOCI library to programmatically interact with a SQL Server DB via ODBC. I am having trouble getting values via output parameters in stored procedures. Some code (modeled after SOCI documentation)...
/*
* create procedure
* Dan.soci_proc #id int, #name varchar(32) output
* as begin
* set nocount on;
* select #name = name from Dan.soci_test where id = #id
* end
*/
std::string sql = "Dan.soci_proc :id, :name";
std::string name;
int proc_ndx = 1;
soci::procedure proc = (my_soci.get_session().prepare << sql, soci::use(proc_ndx),
soci::use(name));
std::cout << "\nAttempting to execute stored procedure " << size << " times."
<< std::endl;
for (; proc_ndx < adj_size; ++proc_ndx) {
try {
proc.execute();
while (proc.fetch())
std::cout << "Fetched: " << name << std::endl;
} catch (const soci::odbc_soci_error& e) {
std::cerr << "Error executing stored procedure." << std::endl;
std::cerr << e.what() << std::endl;
std::cerr << e.odbc_error_message() << std::endl;
return;
}
}
My code does not throw any errors or exceptions, but nothing is fetched, either. I have tried calling it many different ways (normal exec syntax, ODBC call syntax, etc.), but nothing seems to work. I'm wondering if the error goes back to this from here...
If an input/output parameter is omitted, or if a literal is supplied
for the parameter, the driver discards the output value.
Unfortunately, SOCI doesn't really seem to support parameter markers, at least as far as I can tell.
I have made the code work by using this kind of syntax...
std::string sql = "declare #name varchar(32); "
"exec Dan.soci_proc :id, #name = #name output; select #name";
...
soci::procedure proc = (my_soci.get_session().prepare << sql, soci::use(proc_ndx),
soci::into(name));
But that isn't ideal, for reasons that I think are obvious.
Has anyone out there used SOCI and have some input into what I need to do differently to get this to work?
EDIT: I received the following response from the soci-users forum/mailing list...
I haven't used SOCI for a couple of years but I did successfully get
stored procedures to work with ODBC on SQL Server, end of 2011, soci
version 2 (I think).
I remember that to do it I had to use statement rather than procedure.
I had to amend the backend to call SQLMoreResults in a loop.
Output parameters I think were not supported in SOCI and I handled
them direct with ODBC over which I also had a C++ layer.
However, since I'm a recent college grad (just started my first job this month!) and relatively inexperienced with C++ and database applications, I would really appreciate as much help as people are able to offer! For instance, regarding the above response - the last two points are not exactly clear to me.
The problems is that I was using the SOCI ODBC, which apparently does not support stored procedures.
SOCI Existing Backends
SOCI ODBC Backend - Stored Procedures
In the months since I initially asked this question, I have implemented my own SOCI backend, for which stored procedures work perfectly!
I have a C++ library that calls a stored procedure in MSSQL database using ADODB. Everything works fine on development database. But on test database, I am getting Com error 0x800a0e78: This option is not allowed if an object is closed.
The code looks like
ADODB::_CommandPtr cmd = NULL;
ADODB::_RecordsetPtr rs = NULL;
CHECKHR(cmd.CreateInstance( __uuidof( ADODB::Command) ));
cmd->ActiveConnection = m_connexion;
cmd->CommandText = "my_stored_procedure";
cmd->CommandType = ADODB::adCmdStoredProc;
CHECKHR(cmd->Parameters->Refresh());
//input param
cmd->Parameters->GetItem(paramIndex)->put_Value(paramValue);
// output param
cmd->Parameters->GetItem(paramIndex)->PutDirection(ADODB::adParamOutput);
rs = cmd->Execute( NULL, NULL, ADODB::adCmdStoredProc);
while(! rs->ADO_EOF ) { ...
rs->ADO_EOF is where it crashes the program if I use the test database.
note: The stored procedure is same in both the databases and returns same data.
There is one more flow where another SP is called. It works well with the test database. The problem appears only with this particular SP.
I tend to think that it is not a code issue because it works with development database. But I can consistently reproduce the problem with test database.
Please suggest of next actions I should take to resolve this issue
UPDATE
Due to some miracle this C++ exception has gone away after 1 day. But it is so very slow that the execution almost always times out. I do not know how to justify this behavior
UPDATE: 2013-07-18
After so much time, the error has appeared again. This time with development DB with the same SP
[Com error 0x800a0e78 : Operation is not allowed when the object is closed.
on the same line
while(! rs->ADO_EOF ) {
in rs I can see a memory address pointing to ADODB recordset object. But rs->ADO_EOF is generating the said error
I have a C++ program that insert about a million of records into MS Access DB using OLEDBConnection. To do that, I ran the INSERT INTO query a millions time in order to get the records inserted which take quite a long time.
The data is generated in the program in form of array, will that be any other way that i can load the data into database in one single step to improve the performance?
Thanks!
Loop i use to insert the records currently
for (int i = 0; i < populationSize; i++){
insertSQL = "INSERT INTO [" + pTableName + "] (" + columnsName + ") VALUES (" + columnsValue[i] + ");";`
outputDBConn->runSQLEdit(insertSQL);
}
Method that run the SQL query
void DBConnector::runSQLEdit(String^ query){
SQLCMD = gcnew OleDbCommand( query, dbConnection );
SQLCMD->CommandTimeout = 30;
dbConnection->Open();
SQLCMD->ExecuteNonQuery();
dbConnection->Close();
}
It seems very inefficient to open/close the connection for each insert statement.
The standard approach goes something like:
Open connection.
Start transaction, if supported. (This is often very important for databases with transactions.)
Insert. Repeat this step as needed.
Commit transaction, if supported.
Close connection.
Update: The following does not apply to MS Access. Access does not support inserting multiple rows from a literal. It only supports inserting multiple rows from an existing data source. (Although here is a "workabout" that might work. In any case, the most important thing is likely limiting the number of transactions.)
One more thing that can be done is to build a single insert command that adds multiple records at once. This can be done with either multiple statements or a multi-record insert (if supported). It may or may not be significantly faster than just the above (depends upon other factors like network latency and database engine) and will likely need to be adapted to fit within the restrictions of the database (e.g. might only be feasible for a few hundred records at once). This should only be considered after proper connection/transaction usage as described above.
I wouldn't be surprised if there we already-made "bulk insert" libraries/modules floating about... and I don't use MS Access so I can only hope that the above suggestions were helpful :-)
Happy coding.
Don't do ONE insertion per command.
Change your code to something like, this:
string strSQLCommand;
for (int i = 0; i < populationSize; i++){
strSQLCommand += "INSERT INTO [" + pTableName + "] (" + columnsName + ") VALUES (" + columnsValue[i] + ");";`
}
outputDBConn->runSQLEdit(strSQLCommand );
I'm not sure what's the max buffer size of the command, so do some checks and then get the best value to do some "breaks" at every X inserts.