Pass Binary string/file content from c++ to node js - c++

I'm trying to pass the content of a binary file from c++ to node using the node-gyp library. I have a process that creates a binary file using the .fit format and I need to pass the content of the file to js to process it. So, my first aproach was to extract the content of the file in a string and try to pass it to node like this.
char c;
std::string content="";
while (file.get(c)){
content+=c;
}
I'm using the following code to pass it to Node
v8::Local<v8::ArrayBuffer> ab = v8::ArrayBuffer::New(args.GetIsolate(), (void*)content.data(), content.size());
args.GetReturnValue().Set(ab);
In node a get an arrayBuffer but when I print the content to a file it is different to the one that show a c++ cout.
How can I pass the binary data succesfully?
Thanks.

Probably the best approach is to write your data to a binary disk file. Write to disk in C++; read from disk in NodeJS.
Very importantly, make sure you specify BINARY MODE.
For example:
myFile.open ("data2.bin", ios::out | ios::binary);
Do not use "strings" (at least not unless you want to uuencode). Use buffers. Here is a good example:
How to read binary files byte by byte in Node.js
var fs = require('fs');
fs.open('file.txt', 'r', function(status, fd) {
if (status) {
console.log(status.message);
return;
}
var buffer = new Buffer(100);
fs.read(fd, buffer, 0, 100, 0, function(err, num) {
...
});
});
You might also find these links helpful:
https://nodejs.org/api/buffer.html
<= Has good examples for specific Node APIs
http://blog.paracode.com/2013/04/24/parsing-binary-data-with-node-dot-js/
<= Good discussion of some of the issues you might face, including "endianness" and "interpreting numbers"
ADDENDUM:
The OP clarified that he's considering using C++ as a NodeJS Add-On (not a standalone C++ program.
Consequently, using buffers is definitely an option. Here is a good tutorial:
https://community.risingstack.com/using-buffers-node-js-c-plus-plus/
If you choose to go this route, I would DEFINITELY download the example code and play with it first, before implementing buffers in your own application.

It depends but for example using redis
Values can be strings (including binary data) of every kind, for
instance you can store a jpeg image inside a value. A value can't be
bigger than 512 MB.
If the file is bigger than 512MB, then you can store it in chunks.
But I wouldnt suggest since this is an in-memory data store
Its easy to implement in both c++ and node.js

Related

Get raw buffer for in-memory dataset in GDAL C++ API

I have generated a GeoTiff dataset in-memory using GDALTranslate() with a /vsimem/ filepath. I need access to the buffer for the actual GeoTiff file to put it in a stream for an external API. My understanding is that this should be possible with VSIGetMemFileBuffer(), however I can't seem to get this to return anything other than nullptr.
My code is essentially as follows:
//^^ GDALDataset* srcDataset created somewhere up here ^^
//psOptions struct has "-b 4" and "-of GTiff" settings.
const char* filep = "/vsimem/foo.tif";
GDALDataset* gtiffData = GDALTranslate(filep, srcDataset, psOptions, nullptr);
vsi_l_offset size = 0;
GByte* buf = VSIGetMemFileBuffer(filep, &size, true); //<-- returns nullptr
gtiffData seems to be a real dataset on inspection, it has all the appropriate properties (number of bands, raster size, etc). When I provide a real filesystem location to GDALTranslate() rather than the /vsimem/ path and load it up in QGIS it renders correctly too.
Looking a the source for VSIGetMemFileBuffer(), this should really only be returning nullptr if the file can't be found. This suggests i'm using it incorrectly. Does anyone know what the correct usage is?
Bonus points: Is there a better way to do this (stream the file out)?
Thanks!
I don't know anything about the C++ API. But in Python, the snippet below is what I sometimes use to get the contents of an in-mem file. In my case mainly VRT's but it shouldn't be any different for other formats.
But as said, I don't know if the VSI-api translate 1-on-1 to C++.
from osgeo import gdal
filep = "/vsimem/foo.tif"
# get the file size
stat = gdal.VSIStatL(filep, gdal.VSI_STAT_SIZE_FLAG)
# open file
vsifile = gdal.VSIFOpenL(filep, 'r')
# read entire contents
vsimem_content = gdal.VSIFReadL(1, stat.size, vsifile)
In the case of a VRT the content would be text, shown with something like print(vsimem_content.decode()). For a tiff it would of course be binary data.
I came back to this after putting in a workaround, and upon swapping things back over it seems to work fine. #mmomtchev suggested looking at the CPL_DEBUG output, which showed nothing unusual (and was silent during the actual VSIGetMemFileBuffer call).
In particular, for other reasons I had to put a GDALWarp call in between calling GDALTranslate and accessing the buffer, and it seems that this is what makes the difference. My guess is that GDALWarp is calling VSIFOpenL internally - although I can't find this in the source - and this does some kind of initialisation for VSIGetMemFileBuffer. Something to try for anyone else who encounters this.

How to write content of an object into a file in c++

I have a code in this format:
srcSAXController control(input_filename.c_str());
std::string output_filename = input_filename;
output_filename = "c-" + output_filename.erase(input_filename.rfind(XML_STR));
std:: ofstream myfile(output_filename.c_str());
coverage_handler handler(i == MAIN_POS ? true : false, output_filename);
control.parse(&handler);
myfile.write((char *)&control, sizeof(control));
myfile.close();
I want the content of object 'control' to be written into my file. How to fix the code above, so that content of the control object is written to the file.
In general you need much more than just writing the bytes of the object to be able to save and reload it.
The problem is named "serialization" and depending on a lot of factors there are several strategies.
For example it's important to know if you need to save and reload the object on the same system or if you may need to reload it on a different system; it's also fundamental to know if the object contains links to other objects, if the link graph is a simple tree or if there are possibly loops, if you need to support versioning etc. etc.
Writing the bytes to disk like the code is doing is not going to work even for something as simple as an object containing an std::string.

how to read binary file data using dlang

I'm trying to read struct data from specific position in a a binary file.
Found out that I can use import std.stdio and its File, however all i seem to find is all about string handling.
I have c-code written data on binary files that compose of several different structs and they all, as far as I understand coding lay in a oneliner. In order to find specific struct I need to, like in old c,
Open file for reading .... (binary read??)
using sizeof and move to startposition of struct data to read
read data (struct.sizeof data) into receivingbuffer and
Close file
Documentation for std.stdio.File # read talks about reading entire or up to size but can't find how to read as below c-code line?
fseek(filehandle, sizeof(firstStructData), SEEK_SET));
read(filehandle, (char *)nextReceivingBuffer, sizeof(nextReceivingBuffer))
Any ideas or hints?
Try File.seek and File.rawRead. They work like their C counterparts, but rawRead determines the read count from the size of the output buffer you hand it.

Correct way to extract array data from binary?

There is a classic way to embed resource files as a C language array into a binary file, so that we can store some external resource files such as .jpeg or .txt files into a binary.
For example, in the header file we can define an array:
const unsigned char xd_data[] = {
77,90,144,0,3,0,0,0,4,0,0,0,255,255,0,0,184,0,0,0,0,0,0,0,64,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,240,0,0,
0,14,31,186,14,0,180,9,205,33,184,1,76,205,33,84,104,105,115,32,112,114,
111,103,114,97,109,32,99,97,110,110,111,116,32,98,101,32,114,117,110,
32,105,110,32,68,79,83,32,109,111,100,101,46,13,13,10,36,0,0,0,0,0,0,
0,66,163,223,218,6,194,177,137,6,194,177,137,6,194,177,137,105,221,187,
137,13,194,177,137,133,222,191,137,3,194,177,137,105,221,181,137,4,194,
177,137,136,202,238,137,4,194,177,137,6,194,176,137,73,194,177,137,133,
202,236,137,13,194,177,137,48,228,187,137,11,194,177,137,193,196,183,
137,7,194,177,137,82,105,99,104,6,194,177,137,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,80,69,0,0,76,1,4,0,65,162,32,86,0,0,0,0,0,0,0,
0,224,0,47,1,11,1,6,0,0,100,0,0,0,74,0,0,0,0,0,0,228,113,0,0,0,16,0,0,
0,128,0,0,0,0,64,0,0,16,0,0,0,2,0,0,4,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,
224,0,0,0,4,0,0,0,0,0,0,2,0,0,0,0,0,16,0,0,16,0,0,0,0,16,0,0,16,0,0,0,
0,0,0,16,0,0,0,0,0,0,0,0,0,0,0,124,140,0,0,140,0,0,0,0,208,0,0,0,16,0
};
which contains the contents of the resource file, and it will be compile into the final binary.
There are lots of tools and tutorials on the web about this old trick, such as: http://www.rowleydownload.co.uk/arm/documentation/index.htm?http://www.rowleydownload.co.uk/arm/documentation/embed.htm, https://www.fourmilab.ch/xd/ and http://gareus.org/wiki/embedding_resources_in_executables#c_include_method.
However, looks like most of these pages are talking about how to embed the data into binary file using C style array.
My question is, what is the correct way to find the start address of the resource files in the compiled binary in order to extract them? I.e., how can I find the start address of xd_data in the compiled binary?
If you mean finding the byte address in the file where a data block starts just like objdump does but programmatically, then you can use the Binary File Descriptor library (BFD), see here and here.
if you stored data for example an image and you want to load it (for printing or what ever you want) then if you have a function (library) that load it from memory, as example void loadResImage(void * mem); just do loadResImage(xd_data), if not but you have a function that load it from the file, in that case save it to a temp file eg:
int fd=open("tmpfile");
int ret=write(fd,xd_data, sizeof(xd_data));
close(fd);
loadImageFile("tmpfile");
but if you want to access the data outside the program itself (hex editor for example, or an other program), in that case you have to add a starting mark and optionally an ending mark or sizeof data. eg:
const unsigned char xd_data[]={
...
'M','A','G','I','C'};
in example above the end of the data is known, you just do a search to find it. same way, play around and find a suitable way to store the size of the data. but beware of the compiler optimizations.

Serializing structs

I'm working on a homework project and i'm trying to store inventory data into a file.
The inventory data size shouldn't be too large cause technically no one is going to really use it.
I need to write these contents to a file:
• Item Description
• Quantity on Hand
• Wholesale Cost
• Retail Cost
• Date Added to Inventory
I am going to need to make an interface that allows me to do this:
• Add new records to the file
• Display any record in the file
• Change any record in the file
Struct would be the easiest way to go about this imo. If I can just figure out how to read / write structs to a file this should be really easy.
If you could provide a small example on how to do this I would really appreciate it.
Thanks!
Ask your teacher, could you use boost library.
If yes, read boost serilization tutorial, it contains a simple examples: http://www.boost.org/doc/libs/1_38_0/libs/serialization/doc/tutorial.html
But if you want understand how to work with files, you should do this works without any help or boost.
If you want works with std::[io]fstreams you should decide what format you will support:
- text - for this case best way define operator<< and operator>> and use them for writing structure to file or reading;
- binary - your structure should be POD ( plain old data ) and doesn't should contain pointers - and you will use read and write streams methods.
example for binary file:
http://www.codeguru.com/forum/showthread.php?t=269648
If you don't mind really low level, you can just bit copy the structs in and out by casting a pointer to the struct to void* and using sizeof() to get the struct length. (IIRC their is a way to dump/read a void buffer to/from a file)
Note this ONLY works if the data has no pointers/references/etc.
I like C's IO better than C++'s so:
typedef struct { int hi; int mon; char[35] dat; } S;
S s;
S arr[22];
int f;
// write
f = open(/* I forget the args*/);
// one
if(sizeof(s) != write(f, &s, sizeof(s))) Error();
// many
if(sizeof(arr) != write(f, arr, sizeof(arr))) Error();
close(f);
// read
f = open(/* I forget the args*/);
// one
if(sizeof(s) != read(f, &s, sizeof(s))) Error();
// many
if(sizeof(arr) != read(f, arr, sizeof(arr))) Error();
close(f);
IOStream library does it
The ofstream class provides the interface to write data to files as output streams.
The ifstream class provides the interface to read data from files as input streams
Edit- Example
I would go with XML; it's structured, it's text based so you can look at it with any text editor.