Handle PostgreSQL transaction errors in GDALVectorTranslate - c++

In c++ I'm using the GDAL library for importing geo-spatial files into Postgres/PostGIS.
The GDAL library will create a table in the Postgres database and insert the data. But I can't figure out how to handle errors during the inserting of data.
I'm using GDALVectorTranslate https://gdal.org/api/gdal_utils.html#gdal__utils_8h_1aa176ae667bc857ab9c6016dbe62166eb
If an Postgres error occurs the error text will be outputted and the program continues to run. I would like to handle these Postgres errors.
An error could be:
ERROR 1: INSERT command for new feature failed.
ERROR: invalid byte sequence for encoding "UTF8": 0xe5 0x20 0x46
For now I let my program count the rows in the destination table and if zero then assume error. But that doesn't work if appending to an existing table.
auto *dst = (GDALDataset *) GDALVectorTranslate(nullptr, pgDs, 1, &sourceDs, opt, &bUsageError);
if (dst == nullptr) {
std::cout << "ERROR! Couldn't create table" << std::endl;
return FALSE;
} else {
OGRLayer *layer = dst->GetLayerByName(altName);
// Here the rows are counted
if (layer->GetFeatureCount() == 0) {
std::cout << "ERROR! Insert failed" << std::endl;
return FALSE;
}
std::cout << " Imported";
return TRUE;
}

You can register your own error handler to log and count the underlying errors:
struct {/*members for handling errors*/} ctx;
static void myErrorHandler(CPLErr e, CPLErrorNum n, const char* msg) {
ctx *myctx = (ctx*)CPLGetErrorHandlerUserData();
/* do something with ctx to log and increment error count */
}
int myTranslateFunc() {
ctx myctx; //+initialization
CPLPushErrorHandlerEx(&myErrorHandler,&myctx);
auto *dst = (GDALDataset *) GDALVectorTranslate(nullptr, pgDs, 1, &sourceDs, opt, &bUsageError);
CPLPopErrorHandler();
//inspect myctx for potential errors
}

Related

Can't get a value from LMDB

I'm trying to store and fetch some data from LMDB. Data seems to be stored, I can see the keys in my database, but it gives me MDB_NOTFOUND when I try to fetch the value with the same ID I have just stored it under.
Database opening
MDB_env* environment;
MDB_dbi main;
MDB_dbi order;
mdb_env_create(&environment);
mdb_env_set_maxdbs(environment, 2);
mdb_env_open(environment, path.toStdString().c_str(), 0, 0664);
int rc;
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
mdb_dbi_open(txn, "main", MDB_CREATE, &main);
mdb_dbi_open(txn, "order", MDB_CREATE | MDB_INTEGERKEY, &order);
mdb_txn_commit(txn);
Insertion
void Core::Archive::addElement(const Shared::Message& message) {
QByteArray ba;
QDataStream ds(&ba, QIODevice::WriteOnly);
message.serialize(ds);
uint64_t stamp = message.getTime().toMSecsSinceEpoch();
const std::string& id = message.getId().toStdString();
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.size();
lmdbKey.mv_data = (uint8_t*)id.c_str();
lmdbData.mv_size = ba.size();
lmdbData.mv_data = (uint8_t*)ba.data();
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
int rc;
rc = mdb_put(txn, main, &lmdbKey, &lmdbData, 0);
if (rc == 0) {
MDB_val orderKey;
orderKey.mv_size = 8;
orderKey.mv_data = (uint8_t*) &stamp;
rc = mdb_put(txn, order, &orderKey, &lmdbKey, 0);
if (rc) {
mdb_txn_abort(txn);
} else {
rc = mdb_txn_commit(txn);
if (rc) {
qDebug() << "A transaction error: " << mdb_strerror(rc);
}
}
} else {
qDebug() << "An element couldn't been added to the archive, skipping" << mdb_strerror(rc);
mdb_txn_abort(txn);
}
}
Fetching
Shared::Message Core::Archive::getElement(const QString& id) {
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.toStdString().size();
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
MDB_txn *txn;
int rc;
mdb_txn_begin(environment, NULL, MDB_RDONLY, &txn);
rc = mdb_get(txn, main, &lmdbKey, &lmdbData);
if (rc) {
qDebug() <<"Get error: " << mdb_strerror(rc);
mdb_txn_abort(txn);
throw NotFound(id.toStdString(), jid.toStdString());
} else {
//it never comes here
}
}
Testing code
Core::Archive ar();
ar.open("Test");
Shared::Message msg1;
msg1.generateRandomId();
msg1.setBody("oldest");
msg1.setTime(QDateTime::currentDateTime().addDays(-7));
Shared::Message msg2;
msg2.generateRandomId();
msg2.setBody("Middle");
msg2.setTime(QDateTime::currentDateTime().addDays(-4));
Shared::Message msg3;
msg3.generateRandomId();
msg3.setBody("newest");
msg3.setTime(QDateTime::currentDateTime());
ar.addElement(msg2);
ar.addElement(msg3);
ar.addElement(msg1);
Shared::Message d0 = ar.getElement(msg1.getId());
My logs show stored keys. I can see the required key, I can even compare it with the requested key if I use cursors to scroll over the whole storage, it even shows they are equal, but mdb_cursor_get or mdb_get constantly give me MDB_NOTFOUND. What am I doing wrong?
I got it. No matter what I put into database, I have to read it as a char*
Had to modify fetching code
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
I had to change it to
lmdbKey.mv_data = (char*)id.toStdString().c_str();
and it worked

MySQL Trouble transitioning from Injection to Paramterization

I have the following code here that executes a query. Originally, I used SQL Injection to return row results. Hearing I should use parametrization, I rearranged my code and read the MySQL docs on how to do so. I'm using MySQL's C library in a C++ application.
However, it's not returning the results.
I know my SQL statement is 100% fine. It has been tested. The only thing I changed was changing %d (injection) to ?, which accepts the player's ID.
This returns -1. It's a SELECT statement though, so maybe it's normal?
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
This returns 2. This is correct. I have two fields being returned.
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
This returns 0 (success)
if (mysql_stmt_store_result(m_stmt))
Finally, this returns null.
m_result = mysql_store_result(&m_conn);
I need m_result so I can read the rows. "mysql_stmt_store_result" sounds similar, but doesn't return MYSQL_RESULT.
m_result = mysql_store_result(&m_conn);
/// <summary>
/// Executes a query.
/// </summary>
/// <param name="query">The query to execute.</param>
/// <returns>Returns true on success, else false.</returns>
bool SQLConnection::executeQuery_New(const char *query)
{
int param_count = 0;
int affected_rows = 0;
// Validate connection.
if (!m_connected)
return false;
// Initialize the statement
m_stmt = mysql_stmt_init(&m_conn);
if (!m_stmt) {
fprintf(stderr, " mysql_stmt_init(), out of memory\n");
return false;
}
// Prepare the statement
if (mysql_stmt_prepare(m_stmt, query, strlen(query))) {
fprintf(stderr, " mysql_stmt_prepare(), INSERT failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the parameter count from the statement
param_count = mysql_stmt_param_count(m_stmt);
if (param_count != m_bind.size()) {
fprintf(stderr, " invalid parameter count returned by MySQL\n");
return false;
}
// Bind buffers
// The parameter binds are stored in std::vector<MYSQL_BIND>
// I need to convert std::vector<MYSQL_BIND> m_bind to MYSQL_BIND *bnd
MYSQL_BIND *bind = new MYSQL_BIND[m_bind.size() + 1];
memset(bind, 0, sizeof(bind) * m_bind.size());
for (int i = 0; i < param_count; i++)
bind[i] = m_bind[i];
if (mysql_stmt_bind_param(m_stmt, &bind[0]))
{
fprintf(stderr, " mysql_stmt_bind_param() failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Execute the query
if (mysql_stmt_execute(m_stmt)) {
fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
//if (affected_rows == -1) {
// fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
// fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
// return false;
//}
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
// Store the result
if (mysql_stmt_store_result(m_stmt))
{
fprintf(stderr, " failed retrieving result\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
int d = mysql_errno(&m_conn);
return false;
}
// This looks similar to the last above statement, but I need m_result. I used mysql_store_result earlier when using injection and it worked fine, but here in this case it returns null.
m_result = mysql_store_result(&m_conn);
// Close the statement
if (mysql_stmt_close(m_stmt)) {
/* mysql_stmt_close() invalidates stmt, so call */
/* mysql_error(mysql) rather than mysql_stmt_error(stmt) */
fprintf(stderr, " failed while closing the statement\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
return false;
}
// Delete bind array
if (bind) {
delete[] bind;
bind = NULL;
}
return true;
}
How I'm adding an int parameter (player's id):
void SQLConnection::addParam(int buffer, enum_field_types type, unsigned long length)
{
MYSQL_BIND bind;
memset(&bind, 0, sizeof(bind));
bind.buffer = (char *)&buffer;
bind.buffer_type = type;
bind.is_null = 0;
bind.length = &length;
m_bind.push_back(bind);
}
My variables and their types:
class SQLConnection
{
private:
MYSQL m_conn;
MYSQL_ROW m_row;
MYSQL_RES *m_result;
char m_errorMessage[ERROR_MSG_MAX];
bool m_connected;
MYSQL_STMT *m_stmt;
std::vector<MYSQL_BIND> m_bind;
int m_fieldCount;
// ...
And finally its calling function at the end of the SQL statement:
...WHERE player_id = ?;");
conn.addParam(m_id, MYSQL_TYPE_LONGLONG, 0);
if (!conn.executeQuery_New(buffer)) {
conn.close();
return "";
}
// Close the connection.
conn.close();
std::string s = conn.getField("max_value_column_name");
The error code I get is 2014:
https://dev.mysql.com/doc/refman/5.7/en/commands-out-of-sync.html
Just for the sake of interest, this is a prior function I used. This worked fine for injection. Using the new function above with parameterization is the one causing the issues.
bool SQLConnection::executeQuery(const char *query)
{
// Validate connection.
if (!m_connected)
return false;
// Execute the query
int status = mysql_query(&m_conn, query);
if (status != 0) {
sprintf(m_errorMessage, "Error: %s", mysql_error(&m_conn));
return false;
}
// Store the result
m_result = mysql_store_result(&m_conn);
return true;
}
After I started having language religious wars in my head about using C# over C++, I thought I'd give one last attempt here. Any help is appreciated.
Edit:
This is how I read in column names prior to parameterization (maybe this code needs to be updated after calling mysql_stmt_store_result(m_stmt)?
std::string SQLConnection::getField(const char *fieldName)
{
MYSQL_FIELD *field = NULL;
unsigned int name_field = 0;
mysql_stmt_data_seek(m_stmt, 0);
mysql_stmt_fetch_column(m_stmt, &bind, 0, 0);
//mysql_data_seek(m_result, 0);
//mysql_field_seek(m_result, 0);
const unsigned int num_fields = mysql_stmt_field_count(m_stmt);
// const unsigned int num_fields = mysql_num_fields(m_result);
std::vector<char *> headers(num_fields);
for (unsigned int i = 0; (field = mysql_fetch_field(m_result)); i++)
{
headers[i] = field->name;
if (strcmp(fieldName, headers[i]) == 0)
name_field = i;
}
while ((m_row = mysql_fetch_row(m_result))) {
return std::string(m_row[name_field]);
}
return "";
}
Edit:
What I'm finding is in this last function there are equivalent functions for statements, like mysql_num_fields() is mysql_stmt_field_count(). I'm thinking these need to be updated because it's using m_stmt and not m_result anymore, which gives reason to update the functions so m_stmt is used. It's not very apparent how to update the second half of the function though.
You may need a better understanding of how stmt works.You can't get the final results by mysql_store_result() when you're using stmt.
You shoud bind several buffers for the statement you're using to accept the result set. You can finish this by mysql_stmt_bind_result(), just like mysql_stmt_bind_param().
Then you can use the buffers bound by mysql_stmt_bind_result() to return row data. You can finish this by mysql_stmt_fetch().
Do the fetch method repeatedly, so you can get the whole result set row by row.
The basic sequence of calls:
mysql_stmt_init
mysql_stmt_prepare
mysql_stmt_bind_param
mysql_stmt_execute
mysql_stmt_bind_result
mysql_stmt_store_result
mysql_stmt_fetch (repeatedly, row by row)
mysql_stmt_free_result
It works for me.
It's a long time since I finished this part of my project, you'd better read the manual carefully and find more examples of stmt.
Sorry for my poor English. Good luck!

fail to retain pointer to memory

This is how i load pdf using Google's PDFIUM library. I receive void*:
doc = FPDF_LoadDocument("media1.pdf", NULL);
Now i want to save this document in vector of FPDF_DOCUMENT which is basically void* that i receive back from above code. This is how i am saving it in vector:
pdfs.push_back(doc);
Problem is when i do out of scope of this method, i dont have access to that memory pointed by doc. I guess when doc goes out of scope, GC frees that memory. I got to know about this Valgrid debugging tool.
doc is declared in same function as command to load pdf is.
I basically want to retain loaded pdf so that i can later access it and render pages from it.
Valgrind output:
==4816== 64 bytes in 1 blocks are definitely lost in loss record 175 of 616
==4816== at 0x4C2A105: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4816== by 0x85175B: FPDFBitmap_Create (in /home/ec2-user/Vid/dist/Debug/GNU-MacOSX/video_creator_mount)
==4816== by 0x4586A1: Blrt::PDFManager::RenderPDFPage(int, int) (PDFManager.cc:159)
Following is method in PDFManager:
std::unique_ptr<Canvas::LoadedPDFInfo> PDFManager::LoadPDF(const std::vector<uint8_t>& data, size_t dataSize)
{
if(!initPDFIUM) {
InitPDFIUM();
currentPDFHandle = 0;
}
FPDF_DOCUMENT doc;
// doc = FPDF_LoadMemDocument(&data[0], dataSize, nullptr);
doc = FPDF_LoadDocument("media1.pdf", NULL);
if (!doc) {
unsigned long err = FPDF_GetLastError();
fprintf(stderr, "Load pdf docs unsuccessful: ");
switch (err) {
case FPDF_ERR_SUCCESS:
fprintf(stderr, "Success");
break;
case FPDF_ERR_UNKNOWN:
fprintf(stderr, "Unknown error");
break;
case FPDF_ERR_FILE:
fprintf(stderr, "File not found or could not be opened");
break;
case FPDF_ERR_FORMAT:
fprintf(stderr, "File not in PDF format or corrupted");
break;
case FPDF_ERR_PASSWORD:
fprintf(stderr, "Password required or incorrect password");
break;
case FPDF_ERR_SECURITY:
fprintf(stderr, "Unsupported security scheme");
break;
case FPDF_ERR_PAGE:
fprintf(stderr, "Page not found or content error");
break;
default:
fprintf(stderr, "Unknown error %ld", err);
}
fprintf(stderr, ".\n");
return nullptr;
}
pdfs.push_back(doc);
//doc = nullptr;
std::unique_ptr<Canvas::LoadedPDFInfo> pdfInfo(new Canvas::LoadedPDFInfo);
pdfInfo->handle = ++currentPDFHandle;
pdfInfo->totalPageNum = FPDF_GetPageCount(doc);
std::cout << "ERROR\n";
return pdfInfo;
}
pdfs is vector declared in PDFManager.h:
namespace Blrt
{
class PDFManager
{
public:
static std::unique_ptr<Canvas::LoadedPDFInfo> LoadPDF(const std::vector<uint8_t>& data, size_t dataSize);
static std::unique_ptr<Canvas::TextureData> RenderPDFPage(int32_t pdfHandle, int32_t pageNum);
static void Dispose();
private:
static void InitPDFIUM();
static void UnsupportedHandler(UNSUPPORT_INFO*, int type);
static bool initPDFIUM;
static int32_t currentPDFHandle;
static std::vector<FPDF_DOCUMENT> pdfs;
static int32_t nextMultipleOf4(size_t num);
};
}
Following is part of a method where i am accessing pdfs to access pdf data loaded using above mentioned method:
std::unique_ptr<Canvas::TextureData> PDFManager::RenderPDFPage(int32_t pdfHandle, int32_t pageNum)
{
if (pdfs[pdfHandle-1]) {
auto pdf = *(pdfs[pdfHandle-1]);
std::cout << pdfHandle << " " << pageNum << " " <<pdfs.size() << "\n";
if (1 <= pageNum && pageNum <= FPDF_GetPageCount(pdf)) {
auto page = FPDF_LoadPage(pdf, pageNum);
if(page) {
auto textPage = FPDFText_LoadPage(page);
if (textPage) {
auto scale = 2.0;
auto maxTexSize = VideoCreator::VideoCreator::MaxTextureSize;
auto orgWidth = FPDF_GetPageWidth(page);
auto orgHeight = FPDF_GetPageHeight(page);
auto targetWidth = orgWidth * scale;
auto targetHeight = orgHeight * scale;
if (targetWidth > targetHeight) {
if (targetWidth > maxTexSize) {
scale = maxTexSize / targetWidth;
}
} else {
if (targetHeight > maxTexSize) {
scale = maxTexSize / targetHeight;
}
}
From the valgrind message, it isn't the doc you're leaking, it's the bitmap you create later. The doc is not cleaned up as there is no gc in c++. You have to close the document yourself.
But, to fix the error, you need to cleanup the bitmap you created in PDFManager.cc:159.

C++ cassandra client select int value

I'm trying to select an integer value but I can't
This is my code:
cassandra_socket = boost::shared_ptr<TSocket>(new TSocket(host, port));
cassandra_transport = boost::shared_ptr<TFramedTransport>(new TFramedTransport(cassandra_socket));
protocol = boost::shared_ptr<TBinaryProtocol>(new TBinaryProtocol(cassandra_transport));
cassandra_client = new CassandraClient(protocol);
try {
cassandra_transport->open();
cassandra_client->set_keyspace("MPS");
ColumnOrSuperColumn csc;
ColumnPath cpath;
cpath.column_family.assign("SubmitResposes_count");
/* This is required - thrift 'feature' */
cpath.__isset.column = true;
cpath.column = "Counter";
cassandra_client->get(csc, "1", cpath,org::apache::cassandra::ConsistencyLevel::ONE);
cout << "Value read is '" << csc.column.value << "'..." << endl;
}
catch (NotFoundException &nf) {
FORCE_TRACE(0, "NOT FOUND EXCEPTION ERROR: %s", nf.what());
} catch (InvalidRequestException &re) {
FORCE_TRACE(0, "INVALID REQUEST ERROR: %s", re.why);
} catch (TException &tx) {
FORCE_TRACE(0, "TEEXCEPTION ERROR: %s", tx.what());
}
it gives me this exception:
InvalidRequest ERROR: Expected 4 or 0 byte int (1)
& this is the table I've created:
create table SubmitResposes_count(
ID int primary key,
Counter bigint);
Your key is an int:
ID int primary key,
but you are querying using a string key:
cassandra_client->get(csc, "1", cpath,org::apache::cassandra::ConsistencyLevel::ONE);
^^^
Cassandra is trying to validate your string as an int, which is causing this exception.

mysql_real_query returns 1, mysql_error returns a NULL string

When I run a first query, mysql_real_query returns 1, from what I understand, this is normal, I have to call mysql_errno() and mysql_error() to get information about the error. But why does mysql_error() return a NULL string?
After running a second query or mysql_ping I receive a violation exception.
Any clue to what's happening ?
Also, should I be using C++ connector in a C++ program or may the C connector (which is what I am using) work as well. Would I have to compile it under my compiler to get it to work?
Here is a part of the code I use, may this help someone pinpoint me the solution. I do not know what is wrong here.
char req[50];
static char *serverOptions_p[] = {
"abc",
"--datadir=C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.5/data/abc",
"--language=C:/Program Files/MySQL/MySQL Server 5.5/share/english",
NULL
};
int numElements = sizeof (serverOptions_p) / sizeof(char *) - 1;
static char *serverGroups_p[] = { "mysql_server", NULL };
if (mysql_library_init(numElements, serverOptions_p, (char **) serverGroups_p)) {
return;
}
d_connectionHandle_p = mysql_init(NULL);
if (d_connectionHandle_p) {
my_bool mb = true;
mysql_options(d_connectionHandle_p, MYSQL_OPT_RECONNECT, &mb);
if (mysql_real_connect(d_connectionHandle_p, server_p, user_p, password_p, database_p, 0, NULL, 0)) {
sprintf(req, "CREATE DATABASE %s", database_p);
mysql_real_query(d_connectionHandle_p, requete, strlen(requete));
}
}
else {
return;
}
int e;
strcpy(req, "SELECT * FROM test");
if ((e = mysql_real_query(d_connectionHandle_p, req, strlen(req))) != 0) { // This is where it returns 1
const char *err = mysql_error(d_connectionHandle_p); // Returns an empty string
if (err)
{
}
}
According to the manual, a non-zero return value means an error occurred:
Return Values
Zero if the statement was successful. Nonzero if an error occurred.