I have just realized that if I have a variable in a MAT-file (big array) and I want to update this variable by appending a new row (suppose a 2D array) using the MATLAB C/C++ API then it is necessary to load the variable into memory, create a copy with the new row and finally overwrite the existing array with the new one.
Because I am working with big/huge arrays I would like to know how to update a variable without loading the whole thing into memory.
Thank you.
You can only add new data to a file at the end of it, there is no way you can "insert data in the middle of a file" (or the beginning). You need to read the file, and write the new data. If the old data is smaller than the new data, there is no way this can be done "in situ", since the new data will overwrite some of the old data.
Assuming you don't necessarily need to write it back to the original file, you could load a portion of the file at a time, and, at the beginning of processing, create a new file that you write the extended data into. Once you have written all the data, you can rename the new file to the original name of the input file, should you wish to.
Related
I've been experimenting with Apache Arrow. I have used the column oriented memory mapped files for many years. In the past, I've used a separate file for each column. Arrow seems to like to store everything in one file. Is there a way to add a new column without rewriting the entire file?
The short answer is probably no.
Arrow's in-memory format & libraries support this. You can add a chunked array to a table by just creating a new table (this should be zero-copy).
However, it appears you are talking about storing tables in files. None of the common file formats in use (parquet, csv, feather) support partitioning a table in this way.
Keep in mind, if you are reading a parquet file, you can specify which column(s) you want to read and it will only read the necessary data. So if your goal is only to support individual column retrieval/query then you can just build one large table with all your columns.
I've been messing around with binary files the last whole week. One thing I can't figure out how to do is selectively delete a record from the binary file.
Is this possible in c++? If not, what are ways in which you can do this.
I'm contemplating, maybe creating a duplicate file without the record inside, but that would seem rather inefficient for very large data files. Any more efficient ways, or any way of duplicating a file efficiently? I''m surprised there's ways to delete files, but not selectively delete from a file.
Unlike data structures in memory, you can't pull out a record from a file. You have to:
Read the contents of the file into an in-memory data structure
Remove the relevant data from the data structure in-memory.
Write the updated data structure back to the file.
Just make sure ever record have a flag "deleted" which you set to false when you first write the record, but then change to true if you want to mark the record deleted.
I have a stl::map data-structure
key:data pair
which I need to store in a binary file.
key is an unsigned short value, and is not sequential
data is another big structure, but is of fixed size.
This map is managed based on some user actions of add, modify or delete. And I have to keep the file updated every time I update the map. This is to survive a system crash scenario.
Adding can always be done at the end of the file. But, user can modify or delete any of the existing records.
That means I have to randomly access the file to update that modified/deleted record.
My questions are:
Is there a way I can reach the modified record in the file directly without sequentially searching thru the whole records ? ( Max record size is 5000)
On a delete, how do I remove it from the file and move the next record to the deleted record's position ?
Appreciate your help!
Assuming you have no need for the tree structure of std::map and you just need an associative container, the most common way I've seen to do this is to have two files: One with the keys and one with the data. In the key file, it will contain all of they keys along with the corresponding offset of their data in the data file. Since you said the data is all of the same size, updating should be easy to do (since it won't change any of the offsets). Adding is done by appending. Deleting is the only hard part; you can delete the key to remove it from the database, but it's up to you if you want to keep track of "freed" data sections and try to write over them. To keep track of the keys, you might want another associative container (map or unordered_map) in memory with the location of keys in the key file.
Edit: For example, the key file might be (note that offsets are in bytes)
key1:0
key2:5
and the corresponding data file would be
data1data2
This is a pretty tried and true pattern, used in everyone from hadoop to high speed local databases. To get an idea of persistence complications you might consider, I would highly recommend reading this Redis blog, it taught me a lot about persistence when I was dealing with similar issues.
I have several (1-5) very wide (~50,000 columns) .csv files. The files are (.5GB-1GB) in size (avg. size around 500MB). I need to perform a join on the files on a pre-specified column. Efficiency is, of course, the key. Any solutions that can be scaled out to efficiently allow multiple join columns is a bonus, though not currently required. Here are my inputs:
-Primary File
-Secondary File(s)
-Join column of Primary File (name or col. position)
-Join column of Secondary File (name or col. position)
-Left Join or Inner Join?
Output = 1 File with results of the multi-file join
I am looking to solve the problem using a C-based language, but of course an algorithmic solution would also be very helpful.
Assuming that you have a good reason not to use a database (for all I know, the 50,000 columns may constitute such a reason), you probably have no choice but to clench your teeth and build yourself an index for the right file. Read through it sequentially to populate a hash table where each entry contains just the key column and an offset in the file where the entire row begins. The index itself then ought to fit comfortably in memory, and if you have enough address space (i.e. unless you're stuck with 32-bit addressing) you should memory-map the actual file data so you can access and output the appropriate right rows easily as you walk sequentially through the left file.
Your best bet by far is something like Sqlite, there's C++ bindings for it and it's tailor made for lighting fast inserts and queries.
For the actual reading of the data, you can just go row by row and insert the fields in Sqlite, no need for cache-destroying objects of objects :) As an optimization, you should group up multiple inserts in one statement (insert into table(...) select ... union all select ... union all select ...).
If you need to use C or C++, open the file and load the file directly into a database such as MySQL. The C and C++ languages do not have adequate data table structures nor functionality for manipulating the data. A spreadsheet application may be useful, but may not be able to handle the capacities.
That said, I recommend objects for each field (column). Define a record (file specific) as a collection of fields. Read a text line from a file into a string. Let the record load the field data from the string. Store records into a vector.
Create a new record for the destination file. For each record from the input file(s), load the new record using those fields. Finally, for each record, print the contents of each field with separation characters.
An alternative is to whip up a 2 dimensional matrix of strings.
Your performance bottleneck will be I/O. You may want to read huge blocks of data in. The thorn to the efficiency is the variable record length of a CSV file.
I still recommend using a database. There are plenty of free ones out there, such as MySQl.
It depends on what you mean by "join". Are the columns in file 1 the same as in file 2? If so, you just need a merge sort. Most likely a solution based on merge sort is "best". But I agree with #Blindy above that you should use an existing tool like Sqlite. Such a solution is probably more future proof against changes to the column lists.
Is it possible to access a specific one (using a specific index) in a text file without knowing the size of each record?
If you maintain a separate index of record offsets then you can simply consult it for the appropriate location to seek to. Otherwise, no.
If the records happened to be sorted on a convenient key and you can identify where one record ends and another begins, then you can implement a binary or interpolation search approach. You may be able to add this to your text-file format retrospectively to aid the lookup. Otherwise, you're stuck with serial searches from a position with known index (obviously start of file is one, if you know the total number of records you can work backwards from end of file). You can also consider doing one pass to create an index to allow direct access, or having the file embed a list of offsets that can be easily read.
Check out the dbopen() function. If you pass DB_RECNO as the type parameter you can access variable-length records. These records can be delimited by newlines. Essentially your "database" is a flat text file.
The API will conveniently handle inserts and deletes for you as well.