In c++ I'm using the GDAL library for importing geo-spatial files into Postgres/PostGIS.
The GDAL library will create a table in the Postgres database and insert the data. But I can't figure out how to handle errors during the inserting of data.
I'm using GDALVectorTranslate https://gdal.org/api/gdal_utils.html#gdal__utils_8h_1aa176ae667bc857ab9c6016dbe62166eb
If an Postgres error occurs the error text will be outputted and the program continues to run. I would like to handle these Postgres errors.
An error could be:
ERROR 1: INSERT command for new feature failed.
ERROR: invalid byte sequence for encoding "UTF8": 0xe5 0x20 0x46
For now I let my program count the rows in the destination table and if zero then assume error. But that doesn't work if appending to an existing table.
auto *dst = (GDALDataset *) GDALVectorTranslate(nullptr, pgDs, 1, &sourceDs, opt, &bUsageError);
if (dst == nullptr) {
std::cout << "ERROR! Couldn't create table" << std::endl;
return FALSE;
} else {
OGRLayer *layer = dst->GetLayerByName(altName);
// Here the rows are counted
if (layer->GetFeatureCount() == 0) {
std::cout << "ERROR! Insert failed" << std::endl;
return FALSE;
}
std::cout << " Imported";
return TRUE;
}
You can register your own error handler to log and count the underlying errors:
struct {/*members for handling errors*/} ctx;
static void myErrorHandler(CPLErr e, CPLErrorNum n, const char* msg) {
ctx *myctx = (ctx*)CPLGetErrorHandlerUserData();
/* do something with ctx to log and increment error count */
}
int myTranslateFunc() {
ctx myctx; //+initialization
CPLPushErrorHandlerEx(&myErrorHandler,&myctx);
auto *dst = (GDALDataset *) GDALVectorTranslate(nullptr, pgDs, 1, &sourceDs, opt, &bUsageError);
CPLPopErrorHandler();
//inspect myctx for potential errors
}
Related
I'm trying to store and fetch some data from LMDB. Data seems to be stored, I can see the keys in my database, but it gives me MDB_NOTFOUND when I try to fetch the value with the same ID I have just stored it under.
Database opening
MDB_env* environment;
MDB_dbi main;
MDB_dbi order;
mdb_env_create(&environment);
mdb_env_set_maxdbs(environment, 2);
mdb_env_open(environment, path.toStdString().c_str(), 0, 0664);
int rc;
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
mdb_dbi_open(txn, "main", MDB_CREATE, &main);
mdb_dbi_open(txn, "order", MDB_CREATE | MDB_INTEGERKEY, &order);
mdb_txn_commit(txn);
Insertion
void Core::Archive::addElement(const Shared::Message& message) {
QByteArray ba;
QDataStream ds(&ba, QIODevice::WriteOnly);
message.serialize(ds);
uint64_t stamp = message.getTime().toMSecsSinceEpoch();
const std::string& id = message.getId().toStdString();
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.size();
lmdbKey.mv_data = (uint8_t*)id.c_str();
lmdbData.mv_size = ba.size();
lmdbData.mv_data = (uint8_t*)ba.data();
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
int rc;
rc = mdb_put(txn, main, &lmdbKey, &lmdbData, 0);
if (rc == 0) {
MDB_val orderKey;
orderKey.mv_size = 8;
orderKey.mv_data = (uint8_t*) &stamp;
rc = mdb_put(txn, order, &orderKey, &lmdbKey, 0);
if (rc) {
mdb_txn_abort(txn);
} else {
rc = mdb_txn_commit(txn);
if (rc) {
qDebug() << "A transaction error: " << mdb_strerror(rc);
}
}
} else {
qDebug() << "An element couldn't been added to the archive, skipping" << mdb_strerror(rc);
mdb_txn_abort(txn);
}
}
Fetching
Shared::Message Core::Archive::getElement(const QString& id) {
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.toStdString().size();
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
MDB_txn *txn;
int rc;
mdb_txn_begin(environment, NULL, MDB_RDONLY, &txn);
rc = mdb_get(txn, main, &lmdbKey, &lmdbData);
if (rc) {
qDebug() <<"Get error: " << mdb_strerror(rc);
mdb_txn_abort(txn);
throw NotFound(id.toStdString(), jid.toStdString());
} else {
//it never comes here
}
}
Testing code
Core::Archive ar();
ar.open("Test");
Shared::Message msg1;
msg1.generateRandomId();
msg1.setBody("oldest");
msg1.setTime(QDateTime::currentDateTime().addDays(-7));
Shared::Message msg2;
msg2.generateRandomId();
msg2.setBody("Middle");
msg2.setTime(QDateTime::currentDateTime().addDays(-4));
Shared::Message msg3;
msg3.generateRandomId();
msg3.setBody("newest");
msg3.setTime(QDateTime::currentDateTime());
ar.addElement(msg2);
ar.addElement(msg3);
ar.addElement(msg1);
Shared::Message d0 = ar.getElement(msg1.getId());
My logs show stored keys. I can see the required key, I can even compare it with the requested key if I use cursors to scroll over the whole storage, it even shows they are equal, but mdb_cursor_get or mdb_get constantly give me MDB_NOTFOUND. What am I doing wrong?
I got it. No matter what I put into database, I have to read it as a char*
Had to modify fetching code
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
I had to change it to
lmdbKey.mv_data = (char*)id.toStdString().c_str();
and it worked
I have the following code here that executes a query. Originally, I used SQL Injection to return row results. Hearing I should use parametrization, I rearranged my code and read the MySQL docs on how to do so. I'm using MySQL's C library in a C++ application.
However, it's not returning the results.
I know my SQL statement is 100% fine. It has been tested. The only thing I changed was changing %d (injection) to ?, which accepts the player's ID.
This returns -1. It's a SELECT statement though, so maybe it's normal?
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
This returns 2. This is correct. I have two fields being returned.
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
This returns 0 (success)
if (mysql_stmt_store_result(m_stmt))
Finally, this returns null.
m_result = mysql_store_result(&m_conn);
I need m_result so I can read the rows. "mysql_stmt_store_result" sounds similar, but doesn't return MYSQL_RESULT.
m_result = mysql_store_result(&m_conn);
/// <summary>
/// Executes a query.
/// </summary>
/// <param name="query">The query to execute.</param>
/// <returns>Returns true on success, else false.</returns>
bool SQLConnection::executeQuery_New(const char *query)
{
int param_count = 0;
int affected_rows = 0;
// Validate connection.
if (!m_connected)
return false;
// Initialize the statement
m_stmt = mysql_stmt_init(&m_conn);
if (!m_stmt) {
fprintf(stderr, " mysql_stmt_init(), out of memory\n");
return false;
}
// Prepare the statement
if (mysql_stmt_prepare(m_stmt, query, strlen(query))) {
fprintf(stderr, " mysql_stmt_prepare(), INSERT failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the parameter count from the statement
param_count = mysql_stmt_param_count(m_stmt);
if (param_count != m_bind.size()) {
fprintf(stderr, " invalid parameter count returned by MySQL\n");
return false;
}
// Bind buffers
// The parameter binds are stored in std::vector<MYSQL_BIND>
// I need to convert std::vector<MYSQL_BIND> m_bind to MYSQL_BIND *bnd
MYSQL_BIND *bind = new MYSQL_BIND[m_bind.size() + 1];
memset(bind, 0, sizeof(bind) * m_bind.size());
for (int i = 0; i < param_count; i++)
bind[i] = m_bind[i];
if (mysql_stmt_bind_param(m_stmt, &bind[0]))
{
fprintf(stderr, " mysql_stmt_bind_param() failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Execute the query
if (mysql_stmt_execute(m_stmt)) {
fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
//if (affected_rows == -1) {
// fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
// fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
// return false;
//}
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
// Store the result
if (mysql_stmt_store_result(m_stmt))
{
fprintf(stderr, " failed retrieving result\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
int d = mysql_errno(&m_conn);
return false;
}
// This looks similar to the last above statement, but I need m_result. I used mysql_store_result earlier when using injection and it worked fine, but here in this case it returns null.
m_result = mysql_store_result(&m_conn);
// Close the statement
if (mysql_stmt_close(m_stmt)) {
/* mysql_stmt_close() invalidates stmt, so call */
/* mysql_error(mysql) rather than mysql_stmt_error(stmt) */
fprintf(stderr, " failed while closing the statement\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
return false;
}
// Delete bind array
if (bind) {
delete[] bind;
bind = NULL;
}
return true;
}
How I'm adding an int parameter (player's id):
void SQLConnection::addParam(int buffer, enum_field_types type, unsigned long length)
{
MYSQL_BIND bind;
memset(&bind, 0, sizeof(bind));
bind.buffer = (char *)&buffer;
bind.buffer_type = type;
bind.is_null = 0;
bind.length = &length;
m_bind.push_back(bind);
}
My variables and their types:
class SQLConnection
{
private:
MYSQL m_conn;
MYSQL_ROW m_row;
MYSQL_RES *m_result;
char m_errorMessage[ERROR_MSG_MAX];
bool m_connected;
MYSQL_STMT *m_stmt;
std::vector<MYSQL_BIND> m_bind;
int m_fieldCount;
// ...
And finally its calling function at the end of the SQL statement:
...WHERE player_id = ?;");
conn.addParam(m_id, MYSQL_TYPE_LONGLONG, 0);
if (!conn.executeQuery_New(buffer)) {
conn.close();
return "";
}
// Close the connection.
conn.close();
std::string s = conn.getField("max_value_column_name");
The error code I get is 2014:
https://dev.mysql.com/doc/refman/5.7/en/commands-out-of-sync.html
Just for the sake of interest, this is a prior function I used. This worked fine for injection. Using the new function above with parameterization is the one causing the issues.
bool SQLConnection::executeQuery(const char *query)
{
// Validate connection.
if (!m_connected)
return false;
// Execute the query
int status = mysql_query(&m_conn, query);
if (status != 0) {
sprintf(m_errorMessage, "Error: %s", mysql_error(&m_conn));
return false;
}
// Store the result
m_result = mysql_store_result(&m_conn);
return true;
}
After I started having language religious wars in my head about using C# over C++, I thought I'd give one last attempt here. Any help is appreciated.
Edit:
This is how I read in column names prior to parameterization (maybe this code needs to be updated after calling mysql_stmt_store_result(m_stmt)?
std::string SQLConnection::getField(const char *fieldName)
{
MYSQL_FIELD *field = NULL;
unsigned int name_field = 0;
mysql_stmt_data_seek(m_stmt, 0);
mysql_stmt_fetch_column(m_stmt, &bind, 0, 0);
//mysql_data_seek(m_result, 0);
//mysql_field_seek(m_result, 0);
const unsigned int num_fields = mysql_stmt_field_count(m_stmt);
// const unsigned int num_fields = mysql_num_fields(m_result);
std::vector<char *> headers(num_fields);
for (unsigned int i = 0; (field = mysql_fetch_field(m_result)); i++)
{
headers[i] = field->name;
if (strcmp(fieldName, headers[i]) == 0)
name_field = i;
}
while ((m_row = mysql_fetch_row(m_result))) {
return std::string(m_row[name_field]);
}
return "";
}
Edit:
What I'm finding is in this last function there are equivalent functions for statements, like mysql_num_fields() is mysql_stmt_field_count(). I'm thinking these need to be updated because it's using m_stmt and not m_result anymore, which gives reason to update the functions so m_stmt is used. It's not very apparent how to update the second half of the function though.
You may need a better understanding of how stmt works.You can't get the final results by mysql_store_result() when you're using stmt.
You shoud bind several buffers for the statement you're using to accept the result set. You can finish this by mysql_stmt_bind_result(), just like mysql_stmt_bind_param().
Then you can use the buffers bound by mysql_stmt_bind_result() to return row data. You can finish this by mysql_stmt_fetch().
Do the fetch method repeatedly, so you can get the whole result set row by row.
The basic sequence of calls:
mysql_stmt_init
mysql_stmt_prepare
mysql_stmt_bind_param
mysql_stmt_execute
mysql_stmt_bind_result
mysql_stmt_store_result
mysql_stmt_fetch (repeatedly, row by row)
mysql_stmt_free_result
It works for me.
It's a long time since I finished this part of my project, you'd better read the manual carefully and find more examples of stmt.
Sorry for my poor English. Good luck!
This is how i load pdf using Google's PDFIUM library. I receive void*:
doc = FPDF_LoadDocument("media1.pdf", NULL);
Now i want to save this document in vector of FPDF_DOCUMENT which is basically void* that i receive back from above code. This is how i am saving it in vector:
pdfs.push_back(doc);
Problem is when i do out of scope of this method, i dont have access to that memory pointed by doc. I guess when doc goes out of scope, GC frees that memory. I got to know about this Valgrid debugging tool.
doc is declared in same function as command to load pdf is.
I basically want to retain loaded pdf so that i can later access it and render pages from it.
Valgrind output:
==4816== 64 bytes in 1 blocks are definitely lost in loss record 175 of 616
==4816== at 0x4C2A105: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4816== by 0x85175B: FPDFBitmap_Create (in /home/ec2-user/Vid/dist/Debug/GNU-MacOSX/video_creator_mount)
==4816== by 0x4586A1: Blrt::PDFManager::RenderPDFPage(int, int) (PDFManager.cc:159)
Following is method in PDFManager:
std::unique_ptr<Canvas::LoadedPDFInfo> PDFManager::LoadPDF(const std::vector<uint8_t>& data, size_t dataSize)
{
if(!initPDFIUM) {
InitPDFIUM();
currentPDFHandle = 0;
}
FPDF_DOCUMENT doc;
// doc = FPDF_LoadMemDocument(&data[0], dataSize, nullptr);
doc = FPDF_LoadDocument("media1.pdf", NULL);
if (!doc) {
unsigned long err = FPDF_GetLastError();
fprintf(stderr, "Load pdf docs unsuccessful: ");
switch (err) {
case FPDF_ERR_SUCCESS:
fprintf(stderr, "Success");
break;
case FPDF_ERR_UNKNOWN:
fprintf(stderr, "Unknown error");
break;
case FPDF_ERR_FILE:
fprintf(stderr, "File not found or could not be opened");
break;
case FPDF_ERR_FORMAT:
fprintf(stderr, "File not in PDF format or corrupted");
break;
case FPDF_ERR_PASSWORD:
fprintf(stderr, "Password required or incorrect password");
break;
case FPDF_ERR_SECURITY:
fprintf(stderr, "Unsupported security scheme");
break;
case FPDF_ERR_PAGE:
fprintf(stderr, "Page not found or content error");
break;
default:
fprintf(stderr, "Unknown error %ld", err);
}
fprintf(stderr, ".\n");
return nullptr;
}
pdfs.push_back(doc);
//doc = nullptr;
std::unique_ptr<Canvas::LoadedPDFInfo> pdfInfo(new Canvas::LoadedPDFInfo);
pdfInfo->handle = ++currentPDFHandle;
pdfInfo->totalPageNum = FPDF_GetPageCount(doc);
std::cout << "ERROR\n";
return pdfInfo;
}
pdfs is vector declared in PDFManager.h:
namespace Blrt
{
class PDFManager
{
public:
static std::unique_ptr<Canvas::LoadedPDFInfo> LoadPDF(const std::vector<uint8_t>& data, size_t dataSize);
static std::unique_ptr<Canvas::TextureData> RenderPDFPage(int32_t pdfHandle, int32_t pageNum);
static void Dispose();
private:
static void InitPDFIUM();
static void UnsupportedHandler(UNSUPPORT_INFO*, int type);
static bool initPDFIUM;
static int32_t currentPDFHandle;
static std::vector<FPDF_DOCUMENT> pdfs;
static int32_t nextMultipleOf4(size_t num);
};
}
Following is part of a method where i am accessing pdfs to access pdf data loaded using above mentioned method:
std::unique_ptr<Canvas::TextureData> PDFManager::RenderPDFPage(int32_t pdfHandle, int32_t pageNum)
{
if (pdfs[pdfHandle-1]) {
auto pdf = *(pdfs[pdfHandle-1]);
std::cout << pdfHandle << " " << pageNum << " " <<pdfs.size() << "\n";
if (1 <= pageNum && pageNum <= FPDF_GetPageCount(pdf)) {
auto page = FPDF_LoadPage(pdf, pageNum);
if(page) {
auto textPage = FPDFText_LoadPage(page);
if (textPage) {
auto scale = 2.0;
auto maxTexSize = VideoCreator::VideoCreator::MaxTextureSize;
auto orgWidth = FPDF_GetPageWidth(page);
auto orgHeight = FPDF_GetPageHeight(page);
auto targetWidth = orgWidth * scale;
auto targetHeight = orgHeight * scale;
if (targetWidth > targetHeight) {
if (targetWidth > maxTexSize) {
scale = maxTexSize / targetWidth;
}
} else {
if (targetHeight > maxTexSize) {
scale = maxTexSize / targetHeight;
}
}
From the valgrind message, it isn't the doc you're leaking, it's the bitmap you create later. The doc is not cleaned up as there is no gc in c++. You have to close the document yourself.
But, to fix the error, you need to cleanup the bitmap you created in PDFManager.cc:159.
I'm trying to select an integer value but I can't
This is my code:
cassandra_socket = boost::shared_ptr<TSocket>(new TSocket(host, port));
cassandra_transport = boost::shared_ptr<TFramedTransport>(new TFramedTransport(cassandra_socket));
protocol = boost::shared_ptr<TBinaryProtocol>(new TBinaryProtocol(cassandra_transport));
cassandra_client = new CassandraClient(protocol);
try {
cassandra_transport->open();
cassandra_client->set_keyspace("MPS");
ColumnOrSuperColumn csc;
ColumnPath cpath;
cpath.column_family.assign("SubmitResposes_count");
/* This is required - thrift 'feature' */
cpath.__isset.column = true;
cpath.column = "Counter";
cassandra_client->get(csc, "1", cpath,org::apache::cassandra::ConsistencyLevel::ONE);
cout << "Value read is '" << csc.column.value << "'..." << endl;
}
catch (NotFoundException &nf) {
FORCE_TRACE(0, "NOT FOUND EXCEPTION ERROR: %s", nf.what());
} catch (InvalidRequestException &re) {
FORCE_TRACE(0, "INVALID REQUEST ERROR: %s", re.why);
} catch (TException &tx) {
FORCE_TRACE(0, "TEEXCEPTION ERROR: %s", tx.what());
}
it gives me this exception:
InvalidRequest ERROR: Expected 4 or 0 byte int (1)
& this is the table I've created:
create table SubmitResposes_count(
ID int primary key,
Counter bigint);
Your key is an int:
ID int primary key,
but you are querying using a string key:
cassandra_client->get(csc, "1", cpath,org::apache::cassandra::ConsistencyLevel::ONE);
^^^
Cassandra is trying to validate your string as an int, which is causing this exception.
When I run a first query, mysql_real_query returns 1, from what I understand, this is normal, I have to call mysql_errno() and mysql_error() to get information about the error. But why does mysql_error() return a NULL string?
After running a second query or mysql_ping I receive a violation exception.
Any clue to what's happening ?
Also, should I be using C++ connector in a C++ program or may the C connector (which is what I am using) work as well. Would I have to compile it under my compiler to get it to work?
Here is a part of the code I use, may this help someone pinpoint me the solution. I do not know what is wrong here.
char req[50];
static char *serverOptions_p[] = {
"abc",
"--datadir=C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.5/data/abc",
"--language=C:/Program Files/MySQL/MySQL Server 5.5/share/english",
NULL
};
int numElements = sizeof (serverOptions_p) / sizeof(char *) - 1;
static char *serverGroups_p[] = { "mysql_server", NULL };
if (mysql_library_init(numElements, serverOptions_p, (char **) serverGroups_p)) {
return;
}
d_connectionHandle_p = mysql_init(NULL);
if (d_connectionHandle_p) {
my_bool mb = true;
mysql_options(d_connectionHandle_p, MYSQL_OPT_RECONNECT, &mb);
if (mysql_real_connect(d_connectionHandle_p, server_p, user_p, password_p, database_p, 0, NULL, 0)) {
sprintf(req, "CREATE DATABASE %s", database_p);
mysql_real_query(d_connectionHandle_p, requete, strlen(requete));
}
}
else {
return;
}
int e;
strcpy(req, "SELECT * FROM test");
if ((e = mysql_real_query(d_connectionHandle_p, req, strlen(req))) != 0) { // This is where it returns 1
const char *err = mysql_error(d_connectionHandle_p); // Returns an empty string
if (err)
{
}
}
According to the manual, a non-zero return value means an error occurred:
Return Values
Zero if the statement was successful. Nonzero if an error occurred.