how to insert binary data to QuestDb using influxdb line protocol?
there is no documentation on how to insert binary filed value
https://questdb.io/docs/reference/api/ilp/columnset-types/#string
well maybe i could do strings
sometable "abc\0\0\0\"defg .. binary data" 154564564456
I am hesitant that inserting as string could ruin binary parts of inserted values.
Is there a binary way to do it, or will it work correctly if i insert values as a string?
will it create a binary column automatically?
Related
The support for line-separated JSON in c++ seems never completed from git log.
example JSON lines:
https://jsonlines.org/examples/
relevant docs in arrow:
https://arrow.apache.org/docs/cpp/json.html#basic-usage
I am trying to compare an arrow table with a line-delimited json [1].
Several approaches I tried:
read the line-delimited json into an arrow::Table. Because we do not know the schema of the json (e.g. what are the keys in the json and what are the types of the values), there does not seem a way to construct the arrow::schema
Convert the arrow::Table to line-delimited json by converting each row of the arrow::Table into a JSON line. arrow::Table is in columnar format so it seems we have to traverse columns to get elements by index.
[1] https://arrow.apache.org/docs/python/json.html
problem: in c++, using sqlite, im looking for a way to insert multiple rows into a table, directly from a string containing multiple "rows".
i would like to "reformulate" the following statement for insertion into a table T
INSERT into T VALUES(row1); // 1st row insertion.
INSERT into T VALUES(row2); // 2nd row insertion.
i am seeking to substitute it with something like
INSERT into T mycsvvals; // std::string mycsvvals contains 2 csv-rows
where mycsvvals is a string read-in from a CSV file, "containing" 2 rows of comma separated values.
i could parse the string first, put contents into an array and loop the inserts. however, im wondering if sqlite already provides an efficient bulk import/insert from a string, instead of but similar to import/insert from a csv file.
please, can you provide any links/info that will allow me to achieve this?
thx
the answer provided by #Shawn was not what i was hoping for in terms of brevity; it is not possible to bulk insert directly from a csv-string, some looping structure is necessary. but, in terms of efficiency, its the recommended way to go. thanks #Shawn. the full "prescription" for "prepared statements" in sqlite is given and examplified here:
http://blog.quibb.org/2010/08/fast-bulk-inserts-into-sqlite/
In my application I am using a postgresql database table with a "text" column to store
pickled python objects.
As database driver I'm using psycopg2 and until now I only passed python-strings (not unicode-objects) to the DB and retrieved strings from the DB. This basically worked fine until I recently decided to make String-handling the better/correct way and added the following construct to my DB-layer:
psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)
This basically works fine everywhere in my application and I'm using unicode-objects where possible now.
But for this special case with the text-column containing the pickled objects it makes troubles. I got it working in my test-system this way:
retrieving the data:
SELECT data::bytea, params FROM mytable
writing the data:
execute("UPDATE mytable SET data=%s", (psycopg2.Binary(cPickle.dumps(x)),) )
... but unfortunately I'm getting errors with the SELECT for some columns in the production-system:
psycopg2.DataError: invalid input syntax for type bytea
This error also happens when I try to run the query in the psql shell.
Basically I'm planning to convert the column from "text" to "bytea", but the error
above also prevents me from doing this conversion.
As far as I can see, (when retrieving the column as pure python string) there are only characters with ord(c)<=127 in the string.
The problem is that casting text to bytea doesn't mean, take the bytes in the string and assemble them as a bytea value, but instead take the string and interpret it as an escaped input value to the bytea type. So that won't work, mainly because pickle data contains lots of backslashes, which bytea interprets specially.
Try this instead:
SELECT convert_to(data, 'LATIN1') ...
This converts the string into a byte sequence (bytea value) in the LATIN1 encoding. For you, the exact encoding doesn't matter, because it's all ASCII (but there is no ASCII encoding).
I have been trying to insert key value pairs in database using leveldb and it works fine with simple strings. However if I want to store multiple attributes for a key or for example use JSON encoding , how can it be done in c++ . In Node.js leveldb package it can be done by specifying the encoding . Really can't figure this out
JSON is just a string, so I'm not completely sure where you're coming from here…
If you have some sort of in-memory representation of JSON you'll need to serialize it before writing it to the database and parse it when reading.
I want to insert BSON object from C++ to mongodb gridfs.
I cannot find useful doc. about c++ gridfs API.
Can you give an example how to insert or update BSON object on gridfs structure in c++.
Suppose that i have a collection given below;
{ "_id" : "123", "data" : [ { "C1" : "value1","C2" : "value1"}] }
How can i insert this as gridfs in mongodb?
P.S.: I tried to insert data as collections, but i got error because the document exceeds the file size limit.(document to be inserted exceeds maxBsonObjectSize)
E.g. in the document above, the "data" array has sometimes more than 500.000 rows and with more columns.
Thanks in advance
The MongoDB C++ driver has a class called GridFS that will allow you to insert documents into your gridded system. GridFS is designed to operate on things abstracted as files, it reads its input as a stream of bytes rather than as some structured thing like a BSON object. You could convert your large BSON objects into strings to store them:
GridFS grid = new GridFS(*client, "database_name");
const char data[] = myLargeBSONObj.toString();
BSONObj result = grid->storeFile(data, sizeof data, "filename");
Or, you can store your large objects in a file and stream them into GridFS instead:
BSONObj result = grid->storeFile("path/to/file", "filename");
GridFS will not allow you to update a file once it has been inserted into the system. There is no safe way within the defined API to allow updates without encountering race conditions. To "update" a GridFS file, you'll have to read it out of the system, alter it in your application, and re-insert it into GridFS as a new file. This is slow and tedious. If you need to perform updates, I suggest that you instead re-design your schema so that your documents are smaller than 16 MB.