I have a system that returns a void* to a memory block. This memory block stores contiguous data records of different types(int,char,double etc.) and gives the number of bytes of each field in each record.I essentially look up the type of the record and get the value of the record. To retrieve all the records, I do
switch(type)
{
case 'int' : \*(int*)(ptr+index)
case 'char': \*(char*)(ptr+index)
}
When I have to go through 300000 records this is taking a lot of time.Is there a faster way to go through all the records?
If a single block can be of multiple types that can only be resolved at runtime, you will have to dispatch to handlers in a switch statement. Note that:
unions are usually used in C for such things to save space
switch statements are very fast and translate to constant-time lookup tables
If I understand your question correctly -- I assume you're going through each record, sortof saying "for each record i, if its type is 'char' then access the record at location i".
If you know how many records you have to access in advance, can't you just cache them all first?
If I'm completely off track forgive me for not understanding your point.
Your comments finally gave enough information to answer the question: you're writing to an ostringstream. Which means that you're probably doing a lot of string manipulation inside your switch. Figure out how to optimize this, and your performance problems should go away.
(to convince yourself this is the case, simply comment out all code that references the stream and run your program again)
Related
I have been reading up on HeapValidate in an existing code and trying to figure out what does it do. The documentation says that it checks whether the heap control structures are in a consistent state. What does that mean?
The heap is a data structure like any other, and in its metadata-state-variables there are certain conditions that should always be true. As a made-up example, the number of children in a heap-tree-node should always be a non-negative number; so if HeapValidate() reads a child-count variable and sees that it is negative, it knows something has gone badly wrong and can flag that block as broken.
You might wonder, assuming Microsoft’s heap code does not have any bugs, how the heap’s metadata might get in to an invalid/“impossible” state like that in the first place. Since the heap’s metadata structures live in the same address space that the user-code has access to, it’s usually the result of buggy user code writing some other data via an invalid pointer that happens to point at a memory location where the heap’s metadata field happens to reside, silently overwriting/corrupting the metadata.
A few months ago I read a book on security practices, and it suggested the following method for protecting our classes from overwriting with e.g. overflows etc.:
first define a magic number and a fixed-size array (can be a simple integer too)
use that array containing the magic number, and place one at the top, and one at the bottom of our class
a function compares these numbers, and if they are equal, and equal to the static variable, the class is ok, return true, else it is corrupt, and return false.
place this function at the start of every other class method, so this will check the validity of the class on function calls
it is important to place this array at the start and the end of the class
At least this is as I remember it. I'm coding a file encryptor for learning purposes, and I'm trying to make this code exception safe.
So, in which scenarios is it useful, and when should I use this method, or is this something totally useless to count on? Does it depend on the compiler or OS?
PS: I forgot the name of the book mentioned in this post, so I cannot check it again, if anyone of you know which one was it please tell me.
What you're describing sounds a Canary, but within your program, as opposed to the compiler. This is usually on by default when using gcc or g++ (plus a few other buffer overflow countermeasures).
If you're doing mutable operations on your class and you want to make sure you don't have side effects, I don't know if having a magic number is very useful. Why rely on a homebrew validity check when there are mothods out there that are more likely to be successful?
Checksums: I think it'd be more useful for you to hash the unencrypted text and add that to the end of the encrypted file. When decrypting, remove the hash and compare the hash(decrypted text) with what it should be.
I think most, if not all, widely used encryptors/decryptors store some sort of checksum in order to verify that the data has not changed.
This type of a canary will partially protect you against a very specific type of overflow attack. You can make it a little more robust by randomizing the canary value every time you run the program.
If you're worried about buffer overflow attacks (and you should be if you are ever parsing user input), then go ahead and do this. It probably doesn't cost too much in speed to check your canaries every time. There will always be other ways to attack your program, and there might even be careful buffer overflow attacks that get around your canary, but it's a cheap measure to take so it might be worth adding to your classes.
I have a question regarding the performance effects when taking into consideration of two possible methods of 'getting' data from a given struct. It is assumed that the 'name' variable is relative to what the value of 'id' is.
Assuming I have a struct and enum as follows,
enum GenericId { NONE, ONE, TWO };
struct GenericTypeDefinition {
GenericId id;
const char name[8];
...
};
Let's say I wanted to get the name of this struct. Quite easy, I could just refer to the instance of the GenericTypeDefinition struct and refer (or point) to the name member. Simple enough.
Now here is where my performance question becomes relevant. Say I need I create hundreds of these instances, all of which will be locked to a certain number of names and a unique id per. These instances will be referred to as a collection of possible 'GenericTypeDefinition's throughout the program. Keep in mind, the value of 'name' is relative to the value of 'id'. My question is, would I be able to save some memory if I implemented a function like follows (and removed the name variable from the struct),
struct GenericTypeDefinition { // 'name' is now removed.
GenericId id;
...
};
const char* Definition_ToString(GenericEnum e) {
switch (e) {
case NONE: return "Nothing";
case ZERO: return "Zero is not nothing.";
...
}
I assume it would because I am freeing up the need to store the string (8 bytes in length) in each struct that I create.
If you would like any clarification please ask, as I have not been able to find much on this.
If I understand what you're asking, you are putting redundant data into your struct. Essentially, you are able to get the name of the struct from the id in the struct. But, you could also store the name directly in the struct.
So, you are right -- not storing the name will save memory, because you won't store the name with every item. The cost is a bit of time. You will need to call a function to give you the name from the id each time you need it. You will have to weigh these tradeoffs to determine which is more important.
The devil is in details. The answer depends on many things. For example, how often such a structure is allocated, how often it is used and how often char name[8]; is used.
If you remove name from the structure, several scenario may happen:
if you have many objects of this type and a good allocator, you will save space.
if you use those objects extensively during some calculus and you use name only from time to time, you will save time thanks to better cache performance.
if you use name extensively for some computation and your function Definition_ToString is just a little bit more complex than the one in your example, you will loose on performance.
However, in my estimation, optimizations like this can speed up program by some small factor only. It may help in cases when you count in microseconds. If your program is desperately slow, look for asymptotically better algorithm.
In most cases compiler will do this job for you. It usually stores all the const string literals in the RO section of the executable.Depending on the optimization level it may even do away with the memory taken up by the char array in the struct. So your executable size will grow,but it won't effect the run time memory.
However since the name is tied to the ID,logically it makes sense to implement the second version,so in future if you want to add a new id,you don't need to do any redundant work.
In your first case, the task of initializing the structs with the proper ID and NAME means that the program will, at the very beginning, copy the literals, this is, the strings (because I assume you initialize the structs with the strings within the code) to another space in RAM memory, to which the char[ ] will point.
Instead, the second case means that the value is read from the program itself (the literals are hard coded in a table somewhere in the deep assembler code), and will return a pointer to it (correct me if the pointer is not to somewhere in the program but the returning const char* is stored as a variable), therefore you do save some memory.
My personal comment is (which you may see it beyond the question's scope), that even though the second alternative may save you some memory, implies that the IDs and NAMEs are hard coded, therefore leaving out any possibility of expansion during runtime (i.e. you want to add more IDs that are received via a console...).
I'm writing a fairly straightforward function that sends an array over to a file descriptor. However, in order to send the data, I need to append a one byte header.
Here is a simplified version of what I'm doing and it seems to work:
void SendData(uint8_t* buffer, size_t length) {
uint8_t buffer_to_send[length + 1];
buffer_to_send[0] = MY_SPECIAL_BYTE;
memcpy(buffer_to_send + 1, buffer, length);
// more code to send the buffer_to_send goes here...
}
Like I said, the code seems to work fine, however, I've recently gotten into the habit of using the Google C++ style guide since my current project has no set style guide for it (I'm actually the only software engineer on my project and I wanted to use something that's used in industry). I ran Google's cpplint.py and it caught the line where I am creating buffer_to_send and threw some comment about not using variable length arrays. Specifically, here's what Google's C++ style guide has to say about variable length arrays...
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Variable-Length_Arrays_and_alloca__
Based on their comments, it appears I may have found the root cause of seemingly random crashes in my code (which occur very infrequently, but are nonetheless annoying). However, I'm a bit torn as to how to fix it.
Here are my proposed solutions:
Make buffer_to_send essentially a fixed length array of a constant length. The problem that I can think of here is that I have to make the buffer as big as the theoretically largest buffer I'd want to send. In the average case, the buffers are much smaller, and I'd be wasting about 0.5KB doing so each time the function is called. Note that the program must run on an embedded system, and while I'm not necessarily counting each byte, I'd like to use as little memory as possible.
Use new and delete or malloc/free to dynamically allocate the buffer. The issue here is that the function is called frequently and there would be some overhead in terms of constantly asking the OS for memory and then releasing it.
Use two successive calls to write() in order to pass the data to the file descriptor. That is, the first write would pass only the one byte, and the next would send the rest of the buffer. While seemingly straightforward, I would need to research the code a bit more (note that I got this code handed down from a previous engineer who has since left the company I work for) in order to guarantee that the two successive writes occur atomically. Also, if this requires locking, then it essentially becomes more complex and has more performance impact than case #2.
Note that I cannot make the buffer_to_send a member variable or scope it outside the function since there are (potentially) multiple calls to the function at any given time from various threads.
Please let me know your opinion and what my preferred approach should be. Thanks for your time.
You can fold the two successive calls to write() in your option 3 into a single call using writev().
http://pubs.opengroup.org/onlinepubs/009696799/functions/writev.html
I would choose option 1. If you know the maximum length of your data, then allocate that much space (plus one byte) on the stack using a fixed size array. This is no worse than the variable length array you have shown because you must always have enough space left on the stack otherwise you simply won't be able to handle your maximum length (at worst, your code would randomly crash on larger buffer sizes). At the time this function is called, nothing else will be using the further space on your stack so it will be safe to allocate a fixed size array.
I have a some large data structure (N > 10,000) that usually only needs to be created once (at runtime), and can be reused many times afterwards, but it needs to be loaded very quickly. (It is used for user input processing on iPhoneOS.) mmap-ing a file seems to be the best choice.
Are there any data structure libraries for C++ (or C)? Something along the line
ReadOnlyHashTable<char, int> table ("filename.hash");
// mmap(...) inside the c'tor
...
int freq = table.get('a');
...
// munmap(...); inside the d'tor.
Thank you!
Details:
I've written a similar class for hash table myself but I find it pretty hard to maintain, so I would like to see if there's existing solutions already. The library should
Contain a creation routine that serialize the data structure into file. This part doesn't need to be fast.
Contain a loading routine that mmap a file into read-only (or read-write) data structure that can be usable within O(1) steps of processing.
Use O(N) amount of disk/memory space with a small constant factor. (The device has serious memory constraint.)
Small time overhead to accessors. (i.e. the complexity isn't modified.)
Assumptions:
Bit representation of data (e.g. endianness, encoding of float, etc.) does not matter since it is only used locally.
So far the possible types of data I need are integers, strings, and struct's of them. Pointers do not appear.
P.S. Can Boost.intrusive help?
You could try to create a memory mapped file and then create the STL map structure with a customer allocator. Your customer allocator then simply takes the beginning of the memory of the memory mapped file, and then increments its pointer according to the requested size.
In the end all the allocated memory should be within the memory of the memory mapped file and should be reloadable later.
You will have to check if memory is free'd by the STL map. If it is, your customer allocator will lose some memory of the memory mapped file but if this is limited you can probably live with it.
Sounds like maybe you could use one of the "perfect hash" utilities out there. These spend some time opimising the hash function for the particular data, so there are no hash collisions and (for minimal perfect hash functions) so that there are no (or at least few) empty gaps in the hash table. Obviously, this is intended to be generated rarely but used frequently.
CMPH claims to cope with large numbers of keys. However, I have never used it.
There's a good chance it only generates the hash function, leaving you to use that to generate the data structure. That shouldn't be especially hard, but it possibly still leaves you where you are now - maintaining at least some of the code yourself.
Just thought of another option - Datadraw. Again, I haven't used this, so no guarantees, but it does claim to be a fast persistent database code generator.
WRT boost.intrusive, I've just been having a look. It's interesting. And annoying, as it makes one of my own libraries look a bit pointless.
I thought this section looked particularly relevant.
If you can use "smart pointers" for links, presumably the smart pointer type can be implemented using a simple offset-from-base-address integer (and I think that's the point of the example). An array subscript might be equally valid.
There's certainly unordered set/multiset support (C++ code for hash tables).
Using cmph would work. It does have the serialization machinery for the hash function itself, but you still need to serialize the keys and the data, besides adding a layer of collision resolution on top of it if your query set universe is not known before hand. If you know all keys before hand, then it is the way to go since you don't need to store the keys and will save a lot of space. If not, for such a small set, I would say it is overkill.
Probably the best option is to use google's sparse_hash_map. It has very low overhead and also has the serialization hooks that you need.
http://google-sparsehash.googlecode.com/svn/trunk/doc/sparse_hash_map.html#io
GVDB (GVariant Database), the core of Dconf is exactly this.
See git.gnome.org/browse/gvdb, dconf and bv
and developer.gnome.org/glib/2.30/glib-GVariant.html