Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I am not a C++ programmer, but this algorithm appeared in an operating manual for a machine I'm using and I'm struggling to make sense of it. I'd like someone to explain some terms within it, or possibly the flow of the entire code, given that I don't have time to learn C in the course of the project.
Waveform files for the machine in question are made up of a number of tags in curly brackets. The checksum is calculated using the WAVEFORM tag, {WAVEFORM-length: #data}.
The "data" consists of a number of bytes represented as hexadecimal numbers. "length" is the number of bytes in the "data", while "start" apparently points to the first byte in the "data".
I've managed to work out some of the terms, but I'm particularly unsure about my interpretation of ((UINT32 *)start)[i]
UINT32 checksum(void *start, UINT32 length)
{
UINT32 i, result = 0xA50F74FF;
for(i=0; i < length/4; i++)
result = result ^ ((UINT32 *)start)[i];
return(result);
}
So from what I can tell, the code does the following:
Take the address of the first byte in the "data" and the length of the "data"
Create a variable called result, which is an unsigned integer A50F74FF
For each byte in the first 25% of the data string, raise "result" to that power (presumably modulo 2^32)
Return result as the value checksum
Am I correct here or have I misread the algorithm on one of the steps? I feel like I can't be correct, because basing a checksum on only part of the data wouldn't spot errors in the later parts of the data.
For each byte in the first 25% of the data string, raise "result" to that power (presumably modulo 2^32)
This is wrong. ^ is the bitwise XOR operation. It does not raise to a power.
Also, about "of the data string". The algorithm iterates the pointed data as if it is an array of UINT32. In fact, if start doesn't point to (an element of) an array of UINT32, then the behaviour of the program is undefined1. It would be much better to declare the argument to be UINT32* in the first place, and not use the explicit cast.
Also, about "For each byte in the first 25% of the data string", the algorithm appears to go through (nearly2) all bytes from start to start + length. length is presumably measured in bytes, and UINT32 is presumably a type that consists of 4 bytes. Thus an array of UINT32 objects of N bytes contains N/4 elements UINT32 of objects. Note that this assumes that the byte is 8 bits wide which is probably an assumption that the manual can make, but keep in mind that it is not an assumption portable to all systems.
1 UB as far as the C++ language is concerned. But, if it's shown in the operating manual of a machine, then perhaps the special compiler for the particular hardware specifies defined behaviour for this. That said, it is also quite possible for the author of the manual to have made a mistake.
2 If length is not divisible by 4, then the remaining 1-3 bytes are not used.
So the pseudocode for this function is roughly like this:
function checksum(DATA)
RESULT = 0xA50F74FF;
for each DWORD in DATA do
RESULT = RESULT xor DWORD
return RESULT
where DWORD is a four-byte integer value.
The function is actually going though (almost) all of the data (not 25%) but it's doing it in 4-byte increments that's why the length which is in bytes is divided by 4.
I'm trying to create a simple DBMS and although I've read a lot about it and have already designed the system, I have some issues about the implementation.
I need to know what's the best method in C++ to use a series of bits whose length will be dynamic. This series of bits will be saved in order to figure out which pages in the files are free and not free. For a single file the number of pages used will be fixed, so I can probably use a bitset for that. However the number of records per page AND file will not be fixed. So I don't think bitset would be the best way to do this.
I thought maybe to just use a sequence of characters, since each character is 1 byte = 8 bits maybe if I use an array of them I would be able to create the bit map that I want.
I never had to manipulate bits at such a low level, so I don't really know if there is some other better method to do this, or even if this method would work at all.
thanks in advance
If you are just wanting the basics on the bit twiddling, the following is one way of doing it using an array of characters.
Assume you have an array for the bits (the length needs to be (totalitems / 8 )):
unsigned char *bits; // this of course needs to be allocated somewhere
You can compute the index into the array and the specific bit within that position as follows:
// compute array position
int pos = item / 8; // 8 bits per byte
// compute the bit within the byte. Could use "item & 7" for the same
// result, however modern compilers will typically already make
// that optimization.
int bit = item % 8;
And then you can check if a bit is set with the following (assumes zero-based indexing):
if ( bits[pos] & ( 1 << bit ))
return 1; // it is set
else
return 0; // it is not set
The following will set a specific bit:
bits[pos] |= ( 1 << bit );
And the following can be used to clear a specific bit:
bits[pos] &= ~( 1 << bit );
I would implement a wrapper class and simply store your bitmap in a linked list of chunks where each chunk would hold a fixed size array (I would use a stdint type like uint32_t to ensure a given number of bits) then you simply add links to your list to expand. I'll leave contracting as an exercise to the reader.
I want to store a 20-dimensional array where each coordinate can have 3 values,
in a minimal amount of memory (2^30 or 1 Gigabyte).
It is not a sparse array, I really need every value.
Furthermore I want the values to be integers of arbirary but fixed precision,
say 256 bits or 8 words
example;
set_big_array(1,0,0,0,1,2,2,0,0,2,1,1,2,0,0,0,1,1,1,2, some_256_bit_value);
and
get_big_array(1,0,0,0,1,2,2,0,0,2,1,1,2,0,0,0,1,1,1,2, &some_256_bit_value);
Because the value 3 is relative prime of 2. its difficult to implement this using
efficient bitwise shift, and and or operators.
I want this to be as fast as possible.
any thoughts?
Seems tricky to me without some compression:
3^20 = 3486784401 values to store
256bits / 8bitsPerByte = 32 bytes per value
3486784401 * 32 = 111577100832 size for values in bytes
111577100832 / (1024^3) = 104 Gb
You're trying to fit 104 Gb in 1 Gb. There'd need to be some pattern to the data that could be used to compress it.
Sorry, I know this isn't much help, but maybe you can rethink your strategy.
There are 3.48e9 variants of 20-tuple of indexes that are 0,1,2. If you wish to store a 256 bit value at each index, that means you're talking about 8.92e11 bits - about a terabit, or about 100GB.
I'm not sure what you're trying to do, but that sounds computationally expensive. It may be reasonable feasible as a memory-mapped file, and may be reasonably fast as a memory-mapped file on an SSD.
What are you trying to do?
So, a practical solution would be to use a 64-bit OS and a large memory-mapped file (preferably on an SSD) and simply compute the address for a given element in the typical way for arrays, i.e. as sum-of(forall-i(i-th-index * 3^i)) * 32 bytes in pseudeo-math. Or, use a very very expensive machine with that much memory, or another algorithm that doesn't require this array in the first place.
A few notes on platforms: Windows 7 supports just 192GB of memory, so using physical memory for a structure like this is possible but really pushing it (more expensive editions support more). If you can find a machine at all that is. According to microsoft's page on the matter the user-mode virtual address space is 7-8TB, so mmap/virtual memory should be doable. Alex Ionescu explains why there's such a low limit on virtual memory despite an apparently 64-bit architecture. Wikipedia puts linux's addressable limits at 128TB, though probably that's before the kernel/usermode split.
Assuming you want to address such a multidimensional array, you must process each index at least once: that means any algorithm will be O(N) where N is the number of indexes. As mentioned before, you don't need to convert to base-2 addressing or anything else, the only thing that matters is that you can compute the integer offset - and which base the maths happens in is irrelevant. You should use the most compact representation possible and ignore the fact that each dimension is not a multiple of 2.
So, for a 16-dimensional array, that address computation function could be:
int offset = 0;
for(int ii=0;ii<16;ii++)
offset = offset*3 + indexes[ii];
return &the_array[offset];
As previously said, this is just the common array indexing formula, nothing special about it. Note that even for "just" 16 dimensions, if each item is 32 bytes, you're dealing with a little more than a gigabyte of data.
Maybe i understand your question wrong. But can't you just use a normal array?
INT256 bigArray[3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3];
OR
INT256 ********************bigArray = malloc(3^20 * 8);
bigArray[1][0][0][1][2][0][1][1][0][0][0][0][1][1][2][1][1][1][1][1] = some_256_bit_value;
etc.
Edit:
Will not work because you would need 3^20 * 8Byte = ca. 25GByte.
The malloc variant is wrong.
I'll start by doing a direct calculation of the address, then see if I can optimize it
address = 0;
for(i=15; i>=0; i--)
{
address = 3*address + array[i];
}
address = address * number_of_bytes_needed_for_array_value
2^30 bits is 2^27 bytes so not actually a gigabyte, it's an eighth of a gigabyte.
It appears impossible to do because of the mathematics although of course you can create the data size bigger then compress it, which may get you down to the required size although it cannot guarantee. (It must fail to some of the time as the compression is lossless).
If you do not require immediate "random" access your solution may be a "variable sized" two-bit word so your most commonly stored value takes only 1 bit and the other two take 2 bits.
If 0 is your most common value then:
0 = 0
10 = 1
11 = 2
or something like that.
In that case you will be able to store your bits in sequence this way.
It could take up to 2^40 bits this way but probably will not.
You could pre-run through your data and see which is the commonly occurring value and use that to indicate your single-bit word.
You can also compress your data after you have serialized it in up to 2^40 bits.
My assumption here is that you will be using disk possibly with memory mapping as you are unlikely to have that much memory available.
My assumption is that space is everything and not time.
You might want to take a look at something like STXXL, an implementation of the STL designed for handling very large volumes of data
You can actually use a pointer-to-array20 to have your compiler implement the index calculations for you:
/* Note: there are 19 of the [3]'s below */
my_256bit_type (*foo)[3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3][3];
foo = allocate_giant_array();
foo[0][1][1][0][2][1][2][2][0][2][1][0][2][1][0][0][2][1][0][0] = some_256bit_value;
Is there any way to check the size of a record in Ocaml? Something like sizeof of C/C++?
Yes:
# Obj.size (Obj.repr (1,2,3,4,5)) ;;
- : int = 5
But for a record type, the size only depends on the type declaration, so you could just infer it from that.
The actual size occupied in memory is the number returned by Obj.size plus one in words. Words are 32 or 64 bit depending which OCaml version you are using. The additional word is used for book-keeping.
Besides Obj module, there is also a Objsize library from Dmitry Grebeniuk ( http://forge.ocamlcore.org/projects/objsize/ ). It allows you to get more detailed info about values and its size.
Does anyone have idea about this problem ,
I have a code that runs nicely for 3000 element mesh
But when I want to run it for 17000 element mesh it shows me this error:
A common block or variable may not exceed 2147483647 bytes
This is some parts of code:
PARAMETER(NDIM=398316702)
integer IA(NDF+1),LPIVTC(NDF),JA(NDIM)
DIMENSION AA(NDIM)
And it shows the error for
DIMENSION AA(NDIM)
Why is it in that way?
This isn't a characteristic of Fortran per se, but rather your particular compiler or operating system. Note that 2147483647 = 2^31 - 1 or 2 gigabits. Probably you have a 32-bit OS and it is unable to allocate larger amounts of memory. As to why you can't get a full 2^32, perhaps something is using a signed rather than unsigned integer to store addresses and one bit is unavailable.