Because I have to use a third-party bootloader which cannot handle varying lengths of lines in an SREC file, I need a way to instrument objcopy to create a SREC file, where every S1 line has the same lengths (if no info from ELF file is provided, it should be filled with FF).
We can assume that the address increments are consistent, so we do not jump higher in addresses than our specified line length. So we do not need to create in-between lines with all FF e.g.
I found out about --srec-len but this only sets the maximum length size.
I would need something that sets a fixed length size, is there something I can use?
A possible solution would be also to transform the earlier generated SREC file from objcopy with a (third-party) tool
You can try the srec_cat utility to do this. Following is an example command.
"srec_cat long.srec −o short.s19 −line-length=46"
Related
I found that WinPrint or Var.WRITE can only write up to 4095 bytes to the file at a time. For larger size structures, the data out of limits will be lost. To avoid this, we can write multiple times according to the member order.
( If we only know the name of a structure and load the elf through T32, we can find it in the symbol list and view all its members. So, can we get the member name of the structure by some T32 command and then log to file according to the name like Var.WRITE #1 StructA.memberName )
WinPrint.<command>
The WinPrint pre-command is used to generate a hardcopy or a file from one command. The numbers of
columns and lines in the window are adapted to the possibilities of the printer. Printer selection can be
executed by the PRinTer command.
Thus, the output can also be re-routed to a file. In the case of some commands, extended parameters are
possible for printing more than one page.
WinPrint.var.view %m.3 %r.3 structName
This command can output all the contents of the structure to a file. Because var.view is not restricted by 4095, all the contents can be saved to a file.
m stands for multiline. It displays the structure elements in multiple line format. If the elements are in a multidimensional array, the numeric parameter defines the number of levels displayed.
r stands for recursive. This optional number defines the depth of recursion to be displayed. The command SETUP.VarPtr defines the valid address range for pointers. The contents of pointers outside this range are not displayed.
I have really big files, which contain data packages. The file itself is simply a really big string, and the packages are seperated with a string "PACK1.0".
Assuming "XXX" is data, a package looks like this :
PACK1.0XXXXXXXXXXXXXXXXXPACK1.0XXXXXXXXXXXXXXPACK1.0XXXXXXXXXX
I am creating a hash map which contains the number of packages, and the bytes where it begins.
Example:
PACKAGE NR | BYTE WHERE IT BEGINS IN THE STREAM
0 | 0
1 | 128
2 | 256
. | .
. | .
If I want package number 5340, I look in the hashmap at which byte the package begins, go to the byte with stream.seekg(POSITION) and parse the package, in theory.
My final problem is: I want to travel trough the file with a slider, with play&pause options. My thought was that the slider has a min=0 and max=packagecount range.
Is this a good way to traverse through a file?
What problems can this cause? What is a better way to do this?
This is my Code for storing the hashmap (this code assumes a package is 128byte long) :
std::map<int, int> THEMAP;
thefile.seekg(0,std::ios::end);
dataLength=thefile.tellg();
thefile.seekg(0,std::ios::beg);
while(position<dataLength)
{
thefile.seekg(0,position);
position=position+128;
packagecount++;
THEMAP.insert(std::make_pair(packagecount,position));
}
This is usuually a case for memory-mapped-io (MMIO). If you are Windows only then use the MapViewOfFile and the other functions in that family. For cross-platform usage I recommend glib's file map functions. What MMIO does is to map a part of a file (or an entire file) into the process' memory space, so you can access it via a simple pointer. You can determine which part of the file and which size of it is mapped, arbitrarily.
A possible strategy for you could be that you, on startup, map a fixed block of the file into memory in a loop, block by block) and search for the first package identifer in each block. This is relatively quick and gives you a first set of markers. On next access you can use this initial set to find the proper part of the file, map this and scan only this part. Of course, you'd store any marker that comes along.
Later, when you scroll through your file you just map the page (can be smaller this time, depending on how much data you need at a certain point in time) and display the needed data. Obviously, the address of the package markers can at the same time be used as start address for the memory mapping.
Nice side effects are that it is completely irrelevant what size the packages are and you can map files of any size, even gigabyte sized files. By using small views on the file the memory requirement of your application can be very small.
I had requirement to read text file but its too large then I decide to only read some lines in this file. Can I use seek method for jump given line? Then I can only read that line because that text file is too large reading whole file is wasting lot of time. If its not possible, any one give better solution for that? (seek to given line and read it) (I know binary text files are reading byte by byte)
ex of my file
event1 0
subevent 1
subevent 2
event2 3
(In my file after one event its display number of lines I want to seek for previous event)
Yes, you can seek to a point in the file then read from there. One possible problem is that if the lines are all different lengths, a random location in the file will have a higher probability of being in a longer line: you're not getting evenly distributed probabilities of different lines. If you really really must have identical probabilities then you need to make at least one pass over the file to find the start of each line - then you can store those offsets in a vector and randomly select a vector element to guide seeking to the line data in the file. If you only care a little bit, then you can perhaps advance a small but random number of lines past the one you initially seek to... that will even the odds a bit, avoids the initial pass, but isn't perfect. hansmaad's comment adds a neat approach too - perfect results with pretty-good performance - but requires that you have all the lines numbered in the file itself.
Unless each line has exactly the same length, you're going to have to scan through it.
If you want to jump around in it, you can scan through it, saving the offset of each line in a container of your choice, and then use that to seek to a specific line.
Assuming that the lines are variable / random length, I don't believe there is any built-in way to jump directly to the start of a particular line. You can seek to an arbitrary byte position in the file. However, this might land anywhere in the beginning / middle / end of a line.
My best suggestion would be to attack the problem in two steps:
First, make a complete pass through the file, byte by byte, searching for the start of each line. Record the byte position of each line and store it into an array, vector, etc. (Basically, you are creating an index that maps from line number to starting position.) Then, when you have this index built up, you can easily jump to a particular line, by looking up the position in your index.
As far as I know, there is no built-in way to seek to a new line without already knowing where the lines are. I can't tell you the best way to achieve your goal, because most of your question details how you're trying to accomplish it, not what it is you're actually trying to accomplish. Therefore, I might go one of two ways with this:
1) If you actually need every last bit of data from the file (there is no metadata or other information that can be discarded):
Someone mentioned scanning through the file, tracking the lines as you go and building an index with it so you can read in one line at a time. This might work, and it would be the way to go if you actually need each line in its entirety, or if you only need the line number and plan on reading in small pieces at a time from there. However, without knowing details about your constraints or requirements, I would not recommend reading in entire lines using this method for one main reason: I have no way of knowing that one line will not itself be too large to load (what if there is only one line in the file?).
Instead, I would simply allocate a buffer of a size that is an appropriate amount to process at a time, and process the file in chunks of that size until you reach the end. You can stream more data in as you go. Without additional details, I can't tell you what that magic number should be, but the size of the largest chunk of information you might need to process is a good starting point as a minimum.
2) If you don't need every last bit of data from the file (you can discard some of the information in it), then you only need some of it. If you only need select pieces of data, then they are easier to find if they are tagged (which is what XML is for). There are lots of free XML parsers, or you can write your own. Then you'd search for tags instead of arbitrary line numbers, and changes to the file that result in the data being in a different location won't affect your ability to find it if it's tagged, as it would if you're just going by line numbers.
if i have both encrypted message and signature (let's say two different size files), i just want to append them and store in a file together but later i will use the same "only one file" to extract the files and verify in my code.
in that way my code will get only one file as an input but can understand which is encrypted file and signature file...
this file will be used in C or C++ program, will be get as an argument.
thanks.
Why don't you just zip (or any other archive method: tar, 7z,...) them together?
Uhm,... I think what you want to do is store two files in one and then extract both files from that one, but please do correct me if I misunderstood.
You could just use zips (gzip, 7zip, whatever) with no compression. If you want to code it yourself, you can store the following information in a file:
offset 0 - 3: a 4-byte integer with the length of the first file.
offset 4 - 7: a 4-byte integer with the length of the second file.
offset 8+: the first file, followed by the second file.
If you need help doing that, ask away!
I want to be able to read from an unsorted source text file (one record in each line), and insert the line/record into a destination text file by specifying the line number where it should be inserted.
Where to insert the line/record into the destination file will be determined by comparing the incoming line from the incoming file to the already ordered list in the destination file. (The destination file will start as an empty file and the data will be sorted and inserted into it one line at a time as the program iterates over the incoming file lines.)
Incoming File Example:
1 10/01/2008 line1data
2 11/01/2008 line2data
3 10/15/2008 line3data
Desired Destination File Example:
2 11/01/2008 line2data
3 10/15/2008 line3data
1 10/01/2008 line1data
I could do this by performing the sort in memory via a linked list or similar, but I want to allow this to scale to very large files. (And I am having fun trying to solve this problem as I am a C++ newbie :).)
One of the ways to do this may be to open 2 file streams with fstream (1 in and 1 out, or just 1 in/out stream), but then I run into the difficulty that it's difficult to find and search the file position because it seems to depend on absolute position from the start of the file rather than line numbers :).
I'm sure problems like this have been tackled before, and I would appreciate advice on how to proceed in a manner that is good practice.
I'm using Visual Studio 2008 Pro C++, and I'm just learning C++.
The basic problem is that under common OSs, files are just streams of bytes. There is no concept of lines at the filesystem level. Those semantics have to be added as an additional layer on top of the OS provided facilities. Although I have never used it, I believe that VMS has a record oriented filesystem that would make what you want to do easier. But under Linux or Windows, you can't insert into the middle of a file without rewriting the rest of the file. It is similar to memory: At the highest level, its just a sequence of bytes, and if you want something more complex, like a linked list, it has to be added on top.
If the file is just a plain text file, then I'm afraid the only way to find a particular numbered line is to walk the file counting lines as you go.
The usual 'non-memory' way of doing what you're trying to do is to copy the file from the original to a temporary file, inserting the data at the right point, and then do a rename/replace of the original file.
Obviously, once you've done your insertion, you can copy the rest of the file in one big lump, because you don't care about counting lines any more.
A [distinctly-no-c++] solution would be to use the *nix sort tool, sorting on the second column of data. It might look something like this:
cat <file> | sort -k 2,2 > <file2> ; mv <file2> <file>
It's not exactly in-place, and it fails the request of using C++, but it does work :)
Might even be able to do:
cat <file> | sort -k 2,2 > <file>
I haven't tried that route, though.
* http://www.ss64.com/bash/sort.html - sort man page
One way to do this is not to keep the file sorted, but to use a separate index, using berkley db (BerkleyDB). Each record in the db has the sort keys, and the offset into the main file. The advantage to this is that you can have multiple ways of sorting, without duplicating the text file. You can also change lines without rewriting the file by appending the changed line at the end, and updating the index to ignore the old line and point to the new one. We used this successfully for multi-GB text files that we had to make many small changes to.
Edit: The code I developed to do this is part of a larger package that can be downloaded here. The specific code is in the btree* files under source/IO.
Try a modifed Bucket Sort. Assuming the id values lend themselves well to it, you'll get a much more efficient sorting algorithm. You may be able to enhance I/O efficiency by actually writing out the buckets (use small ones) as you scan, thus potentially reducing the amount of randomized file/io you need. Or not.
Hopefully, there are some good code examples on how to insert a record based on line number into the destination file.
You can't insert contents into a middle of the file (i.e., without overwriting what was previously there); I'm not aware of production-level filesystems that support it.
I think the question is more about implementation rather than specific algorithms, specifically, handling very large datasets.
Suppose the source file has 2^32 lines of data. What would be an efficent way to sort the data.
Here's how I'd do it:
Parse the source file and extract the following information: sort key, offset of line in file, length of line. This information is written to another file. This produces a dataset of fixed size elements that is easy to index, call it the index file.
Use a modified merge sort. Recursively divide the index file until the number of elements to sort has reached some minimum amount - true merge sort recurses to 1 or 0 elements, I suggest stopping at 1024 or something, this will need fine tuning. Load the block of data from the index file into memory and perform a quicksort on it and then write the data back to disk.
Perform the merge on the index file. This is tricky, but can be done like this: load a block of data from each source (1024 entries, say). Merge into a temporary output file and write. When a block is emptied, refill it. When no more source data is found, read the temporary file from the start and overwrite the two parts being merged - they should be adjacent. Obviously, the final merge doesn't need to copy the data (or even create a temporary file). Thinking about this step, it is probably possible to set up a naming convention for the merged index files so that the data doesn't need to overwrite the unmerged data (if you see what I mean).
Read the sorted index file and pull out from the source file the line of data and write to the result file.
It certainly won't be quick with all that file reading and writing, but is should be quite efficient - the real killer is the random seeking of the source file in the final step. Up to that point, the disk access is usually linear and should therefore be reasonably efficient.