I tried to add special symbols (i.e : æ_ø_å_Æ_Ø_Å_____£€$&#%¿) to Oracle database table VARCHAR2 column and different results noticed in following methods.
Database character set : SELECT * FROM nls_database_parameters WHERE parameter LIKE '%SET';
NLS_NCHAR_CHARACTERSET AL16UTF16
NLS_CHARACTERSET WE8MSWIN1252
SQL used :
INSERT INTO Test(C1) VALUES('æ_ø_å_Æ_Ø_Å_____£€$&#%¿');
Insert Directly via SQL Developer
Result : æ_ø_å_Æ_Ø_Å_____£€$&#%¿
Insert via SQLPlus
Result : ¿¿_¿¿_¿¿_¿¿_¿¿_¿¿_____¿¿¿¿¿$&#%¿¿
Insert via C++ code using SOCI library
Result :
While using database connection string (WINDOWS_1252 charset) : "oracle://service=<service> user=<user> password=<password> charset=178 ncharset=1000"; result was
æ_ø_å_Æ_Ø_Å_____£€$&#%¿
While using database connection string (UTF_8 charset) : "oracle://service=<service> user=<user> password=<password> charset=871ncharset=1000"; result was
æ_ø_å_Æ_Ø_Å_____£€$&#%¿
C++ code used :
std::string dbConnectionString = "oracle://service=<service> user=<user> password=<password> charset=178 ncharset=1000";
soci::session dbCon;
dbCon.open(dbConnectionString.c_str());
soci::statement *cursor = nullptr;
std::string selectString ="INSERT INTO Test(C1) VALUES('æ_ø_å_Æ_Ø_Å_____£€$&#%¿')";
try
{
cursor = new soci::statement(dbCon);
cursor ->alloc();
cursor ->prepare(selectString);
cursor ->define_and_bind();
cursor ->execute(true);
}
catch (soci::soci_error const & e)
{
std::cout <<"ERROR : ." << e.get_error_category() << " : "<<e.what()<< std::endl;
}
What is the reason for this inconsistent behavior?
Related
I've connected my C++ project to MySQL and successfully created a session. I was able to create a Schema. My issue is that when I try to run simple arbitrary queries like USE testSchema SHOW tables; using the MySQL/C++ api, I run into SQL syntax errors. When I run the function directly in the MySQL shell, the query runs perfectly fine.
Here is the full code
const char* url = (argc > 1 ? argv[1] : "mysqlx://pct#127.0.0.1");
cout << "Creating session on " << url << " ..." << endl;
Session sess(url);
{
cout << "Connected!" << endl;
// Create the Schema "testSchema"; This code creates a schema without issue
cout << "Creating Schema..." << endl;
sess.dropSchema("testSchema");
Schema mySchema = sess.createSchema("testSchema");
cout << "Schema Created!" << endl;
// Create the Table "testTable"; This code runs like normal, but the schema doesn't show
cout << "Creating Table with..." << endl;
SqlStatement sqlcomm = sess.sql("USE testSchema SHOW tables;");
sqlcomm.execute();
}
Here is the console output:
Creating session on mysqlx://pct#127.0.0.1 ...
Connected!
Creating Schema...
Schema Created!
Creating Table with...
MYSQL ERROR: CDK Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SHOW tables' at line 1
The error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SHOW tables' at line 1 is a MySQL error that means I have a syntax error in the query, but when I take a closer look at a query, I see there is nothing wrong with it.
I've copied and pasted the code directly from the cpp file into the mysql shell and it runs perfectly. This tells me that something is up with the formatting of how I'm entering the query in the sql() function. But the documentation for the sql() function is really terse and.
Here is the reference to the sql() function: https://dev.mysql.com/doc/dev/connector-cpp/8.0/class_session.html#a2e625b5223acd2a3cbc5c02d653a1426
Can someone please give me some insight on where I'm going wrong? Also here here is the full cpp code for more context:https://pastebin.com/3kQY8THC
Windows 10
Visual Studio 2019
MySQL 8.0 with Connect/C++ X DevAPI
You can do it in two steps:
sess.sql("USE testSchema").execute();
SqlStatement sqlcomm = sess.sql("SHOW tables");
SqlResult res = sqlcomm.execute();
for(auto row : res)
{
std::cout << row.get(0).get<std::string>() << std::endl;
}
Also, you can use the Schema::getTables():
for(auto table : mySchema.getTables())
{
std::cout << table.getName() << std::endl;
}
Keep in mind that the Schema::getTables() doesn't show the Collections created by Schema::createCollection(). There is also a Schema::getCollections():
for(auto collection : mySchema.getCollections())
{
std::cout << collection.getName() << std::endl;
}
I have been picking my brain for a while trying to figure this one out.
The problem I am having is that the function I am using in Oracle returns a BLOB. It's a list of items that are concatenated together using ||.
From the research I have done,
In the QSQLQuery docs it says "Stored procedures that uses the return statement to return values, or return multiple result sets, are not fully supported. For specific details see SQL Database Drivers." - which leads me to believe I may need to switch to a different codebase if Qt cannot handle it yet.
The documentation for the QOCI driver mentions this "Binary Large Objects (BLOBs) can be read and written, but be aware that this process may require a lot of memory. You should use a forward only query to select LOB fields (see QSqlQuery::setForwardOnly())."
I did set
query.setForwardOnly(true);
Before I prepared or executed the statement.
However, I still get this error
QSqlError("6502", "Unable to execute statement", "ORA-06502: PL/SQL: numeric or value error: character string buffer too small\nORA-06512: at line 55\n")
I had to scrub the code a bit, I hope this is still helpful to give context to what i'm trying to accomplish
temp_clob clob;
name varchar2(183) := ?;
start varchar2(16) := ?;
end varchar2(16) := ?;
count integer := ?;
return_val named_redacted_table_object; -- Sorry had to remove this, it's a table with more Date, Varchar, etc
begin
dbms_lob.createtemporary(temp_clob, true);
return_val := package_name.function_name (
set_name => name,
start_time => to_date(start, 'yyyy-mm-dd hh24:mi'),
end_time => to_date(end, 'yyyy-mm-dd hh24:mi'),
max_count => count);
-- In here was a loop that would break apart the removed table object and make it into strings along the following lines
-- '' || return_val(i).name || return_val(i).value || etc
-- and would store these into the CLOB / temp_clob
? := temp_clob;
end;
I could not get something as simple as this to work
begin
? := 'test123';
end;
With the assumption I could at least read this string in Qt.
Here is my code for Qt
QString name = "test";
QSqlQuery query(db);
query.setForwardOnly(true);
query.prepare(sql);
QString test_sql_text = ui->comboBox_test_text->currentText();
qDebug() << name;
query.bindValue(0, name);
query.bindValue(1, "2003-03-14 00:00");
query.bindValue(2, "2005-03-14 23:00");
query.bindValue(3, 2);
query.bindValue(4, QString(""), QSql::Out);
bool query_executed_ok = query.exec();
qDebug() << "did it execute?" << query_executed_ok;
// qDebug() << query.executedQuery();
qDebug() << query.boundValues();
qDebug() << query.lastError();
QSqlRecord rec = query.record();
qDebug() << rec;
int record_count = rec.count();
qDebug() << "Records: " << record_count;
while (query.next()) {
for(int i=0;i<record_count;i++) {
qDebug() << query.isValid() << " - " << rec.fieldName(i) << " " << query.value(i).toString();
}
}
The error posted appears to be from within the Oracle code; ORA.... You have stripped so much it's hard to see what is actually happening, especially the are where the error apparently occurred. But perhaps using Oracle supplied code that is specifically designed to handle CLOBs. Instead of
'' || return_val(i).name ...
Try
dbms_lob.append(temp_clob, to_clob(return_val(i).name))
begin
? := 'test123';
end;
Bind variables are used to assign values to variables. You define your variable in your pl/sql code and assign a value to it at runtime by using a bind variable. In that case pl/sql code will compile correctly.
In your code the bind variable is used to replace the pl/sql variable, not the value, which will fail. Your pl/sql block cannot be compiled because it cannot resolve the "?".
A valid use of bind variables would be
BEGIN
l_xyz := ?;
END;
where you assign the value test123 at runtime.
It took some fiddling, and I realize I gave fairly cryptic code. So thank you to belayer and Koen for taking a shot at my mess.
What I was able to determine and get working for anyone else running into this:
Let me start off by saying I am not sure if this is a bug, or if i'm doing something in a way that was not intended by the designers of QSqlQuery (The class for handling SQL calls).
The call would work in SQL developer and I would see the intended CLOB with all characters. I was unable to get DBMS_Output to work, however, I saw this post saying to reserve space on the string before binding it to the query.
It solves my issue and shows the result in the debug window. However, it presents a new problem. What if the string becomes larger than my hard coded reserve value?
Here's the updated code for that
query.prepare(sql);
QString name= ui->comboBox_name->currentText();
qDebug() << project;
query.bindValue(":name", project);
query.bindValue(":start_date", "2005-03-14 00:00");
query.bindValue(":end_date", "2006-03-14 23:00");
query.bindValue(":max_count", 3);
QString testStr ="*****";
//testStr.truncate(0); //Maybe this works too?
testStr.reserve( 1000000 ); // This did it for sure
qDebug() << testStr.capacity();
query.bindValue(":result_clob", testStr, QSql::Out);
bool query_executed_ok = query.exec();
qDebug() << "did it execute?" << query_executed_ok;
if (query_executed_ok) {
testStr = query.boundValue(":result_clob").toString();
qDebug() << testStr;
} else {
qDebug() << "string is empty";
}
I got the idea to do this, from THIS post.
External Backups/Snapshots for Google Cloud Spanner recommends to use queries with timestamp bounds to create snapshots for export. On the bottom of the Timestamp Bounds documentation it states:
Cloud Spanner continuously garbage collects deleted and overwritten data in the background to reclaim storage space. This process is known as version GC. By default, version GC reclaims versions after they are one hour old. Because of this, Cloud Spanner cannot perform reads at a read timestamp more than one hour in the past.
So any export would need to complete within an hour. A single reader (i.e. select * from table; using timestamp X) would not be able to export the entire table within an hour.
How can multiple parallel readers be implemented in spanner?
Note: It is mentioned in one of the comments that support for Apache Beam is coming, but it looks like that uses a single reader:
/** A simplest read function implementation. Parallelism support is coming. */
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerReadFn.java#L26
Is there a way to do the parallel reader that beam requires today using exising APIs? Or will Beam need to use something that isn't released yet on google spanner?
It is possible to read data in parallel from Cloud Spanner with the BatchClient class. Follow read_data_in_parallel for more information.
If you are looking to export data from Cloud Spanner, I'd recommend you to use Cloud Dataflow (see the integration details here) as it provides higher level abstractions and takes care data processing details, like scaling and failure handling.
Edit 2018-03-30 - The example project has been updated to use the BatchClient offered by Google Cloud Spanner
After the release of the BatchClient for reading/downloading large amounts of data, the example project below has been updated to use the new batch client instead of the standard database client. The basic idea behind the project is still the same: Copy data to/from Cloud Spanner and any other database using standard jdbc functionality. The following code snippet sets the jdbc connection in batch read mode:
if (source.isWrapperFor(ICloudSpannerConnection.class))
{
ICloudSpannerConnection con = source.unwrap(ICloudSpannerConnection.class);
// Make sure no transaction is running
if (!con.isBatchReadOnly())
{
if (con.getAutoCommit())
{
con.setAutoCommit(false);
}
else
{
con.commit();
}
con.setBatchReadOnly(true);
}
}
When the connection is in 'batch read only mode', the connection will use the BatchClient of Google Cloud Spanner instead of the standard database client. When one of the Statement#execute(String) or PreparedStatement#execute() methods are called (as these allow multiple result sets to be returned) the jdbc driver will create a partitioned query instead of a normal query. The results of this partitioned query will be a number of result sets (one per partition) that can be fetched by the Statement#getResultSet() and Statement#getMoreResults(int) methods.
Statement statement = source.createStatement();
boolean hasResults = statement.execute(select);
int workerNumber = 0;
while (hasResults)
{
ResultSet rs = statement.getResultSet();
PartitionWorker worker = new PartitionWorker("PartionWorker-" + workerNumber, config, rs, tableSpec, table, insertCols);
workers.add(worker);
hasResults = statement.getMoreResults(Statement.KEEP_CURRENT_RESULT);
workerNumber++;
}
The result sets that are returned by the Statement#execute(String) are not executed directly, but only after the first call to ResultSet#next(). Passing these result sets to separate worker threads ensures parallel download and copying of the data.
Original answer:
This project was initially created for conversion in the other direction (from a local database to Cloud Spanner), but as it uses JDBC for both source and destination it can also be used the other way around: Converting a Cloud Spanner database to a local PostgreSQL database. Large tables are converted in parallel using a thread pool.
The project uses this open source JDBC driver instead of the JDBC driver supplied by Google. The source Cloud Spanner JDBC connection is set to read-only mode and autocommit=false. This ensures that the connection automatically creates a read-only transaction using the current time as timestamp the first time you execute a query. All subsequent queries within the same (read-only) transaction will use the same timestamp giving you a consistent snapshot of your Google Cloud Spanner database.
It works as follows:
Set the source database to read-only transactional mode.
The convert(String catalog, String schema) method iterates over all
tables in the source database (Cloud Spanner)
For each table the number of records is determined, and depending on the size of the table, the table is copied using either the main thread of the application or by a worker pool.
The class UploadWorker is responsible for the parallel copying. Each worker is assigned a range of records from the table (for example rows 1 to 2,400). The range is selected by a select statement in this format: 'SELECT * FROM $TABLE ORDER BY $PK_COLUMNS LIMIT $BATCH_SIZE OFFSET $CURRENT_OFFSET'
Commit the read-only transaction on the source database after ALL tables have been converted.
Below is a code snippet of the most important parts.
public void convert(String catalog, String schema) throws SQLException
{
int batchSize = config.getBatchSize();
destination.setAutoCommit(false);
// Set the source connection to transaction mode (no autocommit) and read-only
source.setAutoCommit(false);
source.setReadOnly(true);
try (ResultSet tables = destination.getMetaData().getTables(catalog, schema, null, new String[] { "TABLE" }))
{
while (tables.next())
{
String tableSchema = tables.getString("TABLE_SCHEM");
if (!config.getDestinationDatabaseType().isSystemSchema(tableSchema))
{
String table = tables.getString("TABLE_NAME");
// Check whether the destination table is empty.
int destinationRecordCount = getDestinationRecordCount(table);
if (destinationRecordCount == 0 || config.getDataConvertMode() == ConvertMode.DropAndRecreate)
{
if (destinationRecordCount > 0)
{
deleteAll(table);
}
int sourceRecordCount = getSourceRecordCount(getTableSpec(catalog, tableSchema, table));
if (sourceRecordCount > batchSize)
{
convertTableWithWorkers(catalog, tableSchema, table);
}
else
{
convertTable(catalog, tableSchema, table);
}
}
else
{
if (config.getDataConvertMode() == ConvertMode.ThrowExceptionIfExists)
throw new IllegalStateException("Table " + table + " is not empty");
else if (config.getDataConvertMode() == ConvertMode.SkipExisting)
log.info("Skipping data copy for table " + table);
}
}
}
}
source.commit();
}
private void convertTableWithWorkers(String catalog, String schema, String table) throws SQLException
{
String tableSpec = getTableSpec(catalog, schema, table);
Columns insertCols = getColumns(catalog, schema, table, false);
Columns selectCols = getColumns(catalog, schema, table, true);
if (insertCols.primaryKeyCols.isEmpty())
{
log.warning("Table " + tableSpec + " does not have a primary key. No data will be copied.");
return;
}
log.info("About to copy data from table " + tableSpec);
int batchSize = config.getBatchSize();
int totalRecordCount = getSourceRecordCount(tableSpec);
int numberOfWorkers = calculateNumberOfWorkers(totalRecordCount);
int numberOfRecordsPerWorker = totalRecordCount / numberOfWorkers;
if (totalRecordCount % numberOfWorkers > 0)
numberOfRecordsPerWorker++;
int currentOffset = 0;
ExecutorService service = Executors.newFixedThreadPool(numberOfWorkers);
for (int workerNumber = 0; workerNumber < numberOfWorkers; workerNumber++)
{
int workerRecordCount = Math.min(numberOfRecordsPerWorker, totalRecordCount - currentOffset);
UploadWorker worker = new UploadWorker("UploadWorker-" + workerNumber, selectFormat, tableSpec, table,
insertCols, selectCols, currentOffset, workerRecordCount, batchSize, source,
config.getUrlDestination(), config.isUseJdbcBatching());
service.submit(worker);
currentOffset = currentOffset + numberOfRecordsPerWorker;
}
service.shutdown();
try
{
service.awaitTermination(config.getUploadWorkerMaxWaitInMinutes(), TimeUnit.MINUTES);
}
catch (InterruptedException e)
{
log.severe("Error while waiting for workers to finish: " + e.getMessage());
throw new RuntimeException(e);
}
}
public class UploadWorker implements Runnable
{
private static final Logger log = Logger.getLogger(UploadWorker.class.getName());
private final String name;
private String selectFormat;
private String sourceTable;
private String destinationTable;
private Columns insertCols;
private Columns selectCols;
private int beginOffset;
private int numberOfRecordsToCopy;
private int batchSize;
private Connection source;
private String urlDestination;
private boolean useJdbcBatching;
UploadWorker(String name, String selectFormat, String sourceTable, String destinationTable, Columns insertCols,
Columns selectCols, int beginOffset, int numberOfRecordsToCopy, int batchSize, Connection source,
String urlDestination, boolean useJdbcBatching)
{
this.name = name;
this.selectFormat = selectFormat;
this.sourceTable = sourceTable;
this.destinationTable = destinationTable;
this.insertCols = insertCols;
this.selectCols = selectCols;
this.beginOffset = beginOffset;
this.numberOfRecordsToCopy = numberOfRecordsToCopy;
this.batchSize = batchSize;
this.source = source;
this.urlDestination = urlDestination;
this.useJdbcBatching = useJdbcBatching;
}
#Override
public void run()
{
// Connection source = DriverManager.getConnection(urlSource);
try (Connection destination = DriverManager.getConnection(urlDestination))
{
log.info(name + ": " + sourceTable + ": Starting copying " + numberOfRecordsToCopy + " records");
destination.setAutoCommit(false);
String sql = "INSERT INTO " + destinationTable + " (" + insertCols.getColumnNames() + ") VALUES \n";
sql = sql + "(" + insertCols.getColumnParameters() + ")";
PreparedStatement statement = destination.prepareStatement(sql);
int lastRecord = beginOffset + numberOfRecordsToCopy;
int recordCount = 0;
int currentOffset = beginOffset;
while (true)
{
int limit = Math.min(batchSize, lastRecord - currentOffset);
String select = selectFormat.replace("$COLUMNS", selectCols.getColumnNames());
select = select.replace("$TABLE", sourceTable);
select = select.replace("$PRIMARY_KEY", selectCols.getPrimaryKeyColumns());
select = select.replace("$BATCH_SIZE", String.valueOf(limit));
select = select.replace("$OFFSET", String.valueOf(currentOffset));
try (ResultSet rs = source.createStatement().executeQuery(select))
{
while (rs.next())
{
int index = 1;
for (Integer type : insertCols.columnTypes)
{
Object object = rs.getObject(index);
statement.setObject(index, object, type);
index++;
}
if (useJdbcBatching)
statement.addBatch();
else
statement.executeUpdate();
recordCount++;
}
if (useJdbcBatching)
statement.executeBatch();
}
destination.commit();
log.info(name + ": " + sourceTable + ": Records copied so far: " + recordCount + " of "
+ numberOfRecordsToCopy);
currentOffset = currentOffset + batchSize;
if (recordCount >= numberOfRecordsToCopy)
break;
}
}
catch (SQLException e)
{
log.severe("Error during data copy: " + e.getMessage());
throw new RuntimeException(e);
}
log.info(name + ": Finished copying");
}
}
I created a new TYPE in Oracle in order to have parity between my table and a local c++ object (I am using OCCI interface for C++).
In the code I use
void insertRowInTable ()
{
string sqlStmt = "INSERT INTO MY_TABLE_T VALUES (:x)";
try{
stmt = con->createStatement (sqlStmt);
ObjectDefinition *o = new ObjectDefinition ();
o->setA(0);
o->setB(1);
o->setC(2);
stmt->setObject (1, o);
stmt->executeUpdate ();
cout << "Insert - Success" << endl;
delete (o);
}catch(SQLException ex)
{
//exception code
}
The code compiles, connects to db but throws the following exception
Exception thrown for insertRow Error number: 947 ORA-00947: not enough
values
Do I have a problematic "sqlStmt"? Is something wrong with the syntax or the binding?
Of course I have already setup an environment and connection
env = Environment::createEnvironment (Environment::OBJECT);
occiobjm (env);
con = env->createConnection (user, passwd, db);
How many columns are in the table? The error message indicates that you didn't provide enough values in the insert statement. If you only provide a VALUES clause, all columns in the table must be provided. Otherwise you need to list each of the columns you're providing values for:
string sqlStmt = "INSERT INTO MY_TABLE_T (x_col) VALUES (:x)";
Edit:
The VALUES clause is listing placeholder arguments. I think you need to list one for each value passed, e.g.:
string sqlStmt = "INSERT INTO MY_TABLE_T (GAME_ID, VERSION) VALUES (:x1,:x2)"
Have a look at occidml.cpp in the Oracle OCCI docs for an example.
I'm using pqlib with postgresql version 9.1.11
I have the following code
const char *spid = std::to_string(pid).c_str();
PGresult *res;
const char *paramValues[2] = {u->getID().c_str(), spid};
std::string table;
table = table.append("public.\"").append(Constants::USER_PATTERNS_TABLE).append("\"");
std::string param_name_pid = Constants::RELATION_TABLE_PATTERN_ID;
std::string param_name_uid = Constants::RELATION_TABLE_USER_ID;
std::string command = Constants::INSERT_COMMAND + table + " (" + param_name_uid + ", " + param_name_pid + ") VALUES ($1, $2::int)";
std::cout << "command: " << command << std::endl;
res = PQexecParams(conn, command.c_str(), 2, NULL, paramValues, NULL, NULL,0);
Where
INSERT_COMMAND = "INSERT INTO " (string)
USER_PATTERN_TABLE = "User_Patterns" (string)
RELATION_TABLE_PATTERN_ID = "pattern_id" (string)
RELATION_TABLE_USER_ID = "user_id" (string)
pid = an int
u->getID() = a string
conn = the connection to the db
The table "User_Patterns" is defined as
CREATE TABLE "User_Patterns"(
user_id TEXT references public."User" (id) ON UPDATE CASCADE ON DELETE CASCADE
,pattern_id BIGSERIAL references public."Pattern" (id) ON UPDATE CASCADE
,CONSTRAINT user_patterns_pkey PRIMARY KEY (user_id,pattern_id) -- explicit pk
)WITH (
OIDS=FALSE
);
I already have a user and a pattern loaded into their respective tables.
The command generated is :
INSERT INTO public."User_Patterns" (user_id, pattern_id) VALUES ($1, $2::int)
I also tried with $2, $2::bigint, $2::int4
The problem is:
I receive the error :
ERROR: invalid input syntax for integer: "public.""
I already use PQexecParams to store users and patterns, the only difference is that they all have text/xml fields (the only int field on patterns is a serial one and I don't store that value myself) but because the user_patterns is a relation table I need to store and int for the pattern_id.
I already read the docs for pqlib and saw the examples, both are useless.
The problem is in the lines:
const char *spid = std::to_string(pid).c_str();
const char *paramValues[2] = {u->getID().c_str(), spid};
std::to_string(pid) creates temporary string and .c_str() returns a pointer to an internal representation of this string, which is destroyed at the end of the line, resulting in a dead pointer. You may also see answer to the question
stringstream::str copy lifetime