Is it safe to access the same array from multiple goroutines, when every goroutine works on a slice, pointing to the same underlying array but without overlapping ?
Like:
var arr [100]int
sliceA := arr[:50]
sliceB := arr[50:]
go WorkOn(sliceA)
go WorkOn(sliceB)
Just imagine "WorkOn" would do something fancy.
As long as you can guarantee the areas won't overlap, it's fine.
By guarantee I mean: whomever works on sliceA, should not be allowed to do sliceA = append(sliceA, a, b, c). Because then it'll start running into sliceB's territory.
Relevant here, is some documentation for Go 1.2:
This concerns a new language element: 3-index slices:
Go 1.2 adds the ability to specify the capacity as well as the length when using a slicing operation on an existing array or slice. A slicing operation creates a new slice by describing a contiguous section of an already-created array or slice:
var array [10]int
slice := array[2:4]
The capacity of the slice is the maximum number of elements that the slice may hold, even after reslicing; it reflects the size of the underlying array. In this example, the capacity of the slice variable is 8.
Go 1.2 adds new syntax to allow a slicing operation to specify the capacity as well as the length. A second colon introduces the capacity value, which must be less than or equal to the capacity of the source slice or array, adjusted for the origin. For instance,
slice = array[2:4:7]
sets the slice to have the same length as in the earlier example but its capacity is now only 5 elements (7-2). It is impossible to use this new slice value to access the last three elements of the original array.
In this three-index notation, a missing first index ([:i:j]) defaults to zero but the other two indices must always be specified explicitly. It is possible that future releases of Go may introduce default values for these indices.
Further details are in the design document.
Actually jimt's answer MAY be wrong. It depends... :)
E.g. if you are using a []uint8, then a operation like
p[2] = 5
is essentially this
tmp = p[0..3] // this is 32 bit
tmp[2] = 5
p[0..3] = tmp // yeah this is all fake syntax but you'll get it
This is because your CPU is 32 (or even 64) bit. So that is actually more efficient although it seems more complex.
But as you can see, you are WRITING p[0,1,3] although you only intended to write to p[2]. This can create some fun bugs to debug! :)
If your data is e.g. pointers to your data then this issue should not occur as arrays are guaranteed to be stored in memory so that this problem doesn't occur as long as your data is as long as your native instruction set.
Related
This is more of an etiquette question than anything else, but when creating new arrays what value, other than zero, should I use to indicate an empty space in the array? For example:
int* arr;
arr = new int[10];
When I create a new array like in the code above, the array will be filled with ten zeroes. The issue I'm having is that I want to use underscores when printing the array to indicate empty spaces, however, I also have zeroes as part of my data set in the array. So, should I just fill the empty array with some arbitrary value that is unlikely to show up in my data set (like -32000 for example), and use that as the indicator for empty space, or is there some sort of null value that I could use instead so that I can know for a fact that the value at that specific index is definitely an empty space?
What you appear to be asking about is called a sentinel -- some data value that has special meaning.
Regarding choice of sentinel, use something you know is not going to appear and make it a named constant. For example, you might use:
constexpr int NoValue = std::numeric_limits<int>::min();
If you absolutely need the entire integer range or if you cannot reliably sanitize your input to ensure that data is never accepted as a non-empty value, consider using a larger data type that can represent that range and a sentinel, or use std::optional as suggested in another answer.
Alternatively, maintain a separate array to hold that information. Such an array only requires one bit per element to represent whether the value is empty or not and so it only means a fractional increase in storage as opposed to expanding your data type beyond int. This approach trades off memory usage against memory locality, since the data about "emptiness" would not be stored adjacent to a value in your array and that may have implications for caching.
Regarding the actual initialization question: your array is uninitialized and will require setting values with std::fill or similar. Otherwise your program's behavior is undefined if you attempt to use an uninitialized value. Note that there's a special case: new int[100]() which will zero-initialize the memory. But you can't use that construction to initialize with any other value.
Consider using std::vector to avoid memory management issues, and to provide initialization with non-zero values without adding code clutter:
std::vector<int> arr(10, NoValue);
As you can see, there are choices to be made which depend on your program's requirements and its input specification. I hope this helps you make a more informed decision.
should I just fill the empty array with some arbitrary value that is unlikely to show up in my data set
Well, unlikely is not the same as a value that you know for certain will not appear in the data, and the error that you get from being wrong about this will be a nasty error. Generally speaking, however, you usually do have some idea of the range of valid values and it can indeed be easier to use a sentinel value outside that range to indicate nullity. (And if you do it this way I'd recommend being very fastidious about sanitizing the input data coming into your program i.e. explicitly test for your sentinel value unexpectedly coming in from an external source.)
However, in cases where there is no such value, or just to unambiguously declare your intent, the canonical way to handle this situation in modern C++ is to use std::optional<int>. The standard library's optional is a way of turning any type into a nullable type.
Please note that questions of "taste" are generally frowned upon in StackOverflow.
With that said, here's my preference:
Something that cannot masquerade as a valid value, like NaN, makes a good placeholder. If that isn't an option, then as you said, a value that will not/is not allowed to appear in the data set works.
I know that I can remove something in the middle of an array such as
char* cArray = (char*) malloc(sizeof(char) * sizeTracker);
using the memmove function. In this way something will be removed from the array without having to use a temp array, switching to vectors, etc. The question here is that can I add a new index in the middle of the array (is there a function for it)? Or let's say that using realloc I add a new index at the end, then how can I move the values down efficiently?
Alternative Answer
I have been thinking about this and the comments where #DietmarKühl started talking about inserting blocks like a deque does. The problem with this is that a deque is a linked list of blocks so then you can't start with an array. if you start with an array and then want to insert something in the middle you have to do something else and I think I have an idea - it isn't fleshed out very much so it may not work but I will share it anyway. Please leave comments telling me what you think of the idea.
If you had an array of items and then want to add an item into the middle all you really want to do is add a block and update the mapping. The mapping is the thing that makes it all work - but it slows down access because you need to check the mapping before every access of the array.
The mapping would be a binary tree. It would start empty but the nodes would contain a value: if the index you want is < the value you traverse the left pointer and if it is >= you traverse the right pointer.
So, an example:
Before the insert:
root -> (array[100000], offset: 0)
After the insert at 5000:
root -> {value: 5000,
left: (array[100000], offset: 0),
right: {value: 5001,
left: (newarray[10], offset: -5000),
right: (array[100000], offset: 1),
}
}
I have used blocks of 10 here - newarray is 10 in size. If you just randomly insert indexes all over the place the block size should be 1 but if you insert groups of consecutive indexes having a blovk size larger than 1 would be good. It really depends on your usage pattern...
When you check index 7000 you check the root node: 7000 is >= 5000 so you follow the right pointer: 7000 is >= 5001 so you follow the right pointer: it points to the original array with an offset of 1 so you access array[index+offset].
When you check index 700 you check the root node: 700 is < 5000 so you follow the left pointer: it points to the original array with an offset of 0 so you access array[index+offset].
When you check index 5000 you check the root node: 5000 is >= 5000 so you follow the right pointer: 5000 is < 5001 so you follow the left pointer: it points to the new array with an offset of -5000 so you access newarray[index+offset].
Of course optimizations to this would be really important to make this useful - you would have to balance the tree after each insert because otherwise the right side would be much much longer than the left side.
The downside to this is that accesses to the array are now O(log inserts) instead of O(1) so if there are lots of inserts you will want to realloc every so often to compact the data structure back to an array but you could save that for an opportune time.
Like I said it isn't very fleshed out so it may not work in practice but I hope it is worth sharing anyway.
Original Answer
If you have a C style array and want to insert an index in the middle you would need to either have an array larger than you need (plus a variable like sizeTracker to keep track of the size).
Then if there was room left you could just memmove the last half of the array out one to create a spot in the middle.
If there wasn't any room left you could malloc another whole array that includes extra space and then memmove the first half and memmove the second half separately leaving a gap.
If you want to make the malloc amortized constant time you need to double the size of the array each time you reallocate it. The memmove becomes one machine instruction on x86 but even then it will still be O(n) because of moving every value.
But performance isn't any worse then your deleting trick - if you can delete everywhere throughout the array the cost is O(n) for that as well because you memmove half the values in average when you delete.
There is no custom C function which allows to increase an array using the C memory function and inserting an object into the middle. Essentially you'd build the functionality using malloc(), free(), memmove() (when enough space is available and elements are just moved back within the memory), or memcpy() (if you need to allocate new memory and you want to avoid first copying and then moving the tail).
In C++ where object locations tend to matter you'd obviously use std::copy(), std::reverse_copy() and/or std::move() (both forms thereof) as there may be relevant structors for the respect objects. Most likely you'd also obtain memory different, e.g., using operator new() and/or an allocator if you really travel in terms of raw memory.
The fun implementation of the actual insertion (assuming there is enough space for another element) is using std::rotate() to construct the last element and then shuffle elements:
void insert(T* array, std::size_t size, T const& value) {
// precodition: array points to at least size+1 elements
new(array + size) T(value);
std::rotate(array, array + size, array + size + 1);
}
Of course, this doesn't avoid potentially unnecessarily shuffling elements when the array needs to be relocated. In that case it more effective to allocate new memory and move the initial objects to the start, add the newly inserted element, move the trailing objects to the location right past the new object.
If you are using manually allocated memory you have to reallocate and you should hope this operation does not move the memory block to a new location. Then the best is to use the rotate algorithm.
By the way, prefer stl containers such as vectors to manually allocated memory for this kind of tasks. If you are using vectors you should have reserved memory.
You have marked this post as C++.
can I add a new index in the middle of the array (is there a function
for it)
No. From cppreference.com, std::array:
std::array is a container that encapsulates fixed size arrays.
I interpret this to mean you can change the elements, but not the indexes.
(sigh) But I suspect C style arrays are still allowed.
And I notice Dietmar's answer also says no.
Say you have a buffer of size N which must be set to definite values (say to zero, or something else). This value setting in the buffer is divided over M threads, each handling N / M elements of the buffer.
The buffer cannot be immutable, since we change the values. Message passing won't work either, since it is forbidden to pass ref or array (= pointer) types. So it must happen through shared? No, since in my case the buffer elements are of type creal and thus arithmetics are not atomic.
At the end, the main program must wait until all threads are finished. It is given that each thread only writes to a subset of the array and none of the threads have overlap in the array with another thread or in any way depend on eachother.
How would I go about writing to (or modifying) a buffer in a concurrent manner?
PS: sometimes I can simply divide the array in M consecutive pieces, but sometimes I go over the array (the array is 1D but represents 2D data) column-by-column. Which makes the individual arrays the threads use be actually interleaved in the mother-array. Argh.
EDIT: I figured out that the type shared(creal)[] would work, since now the elements are shared and not the array itself. You could parallelize interleaved arrays I bet. There is some disadvantage though:
The shared storage class is so strict, that the allocation must be supplied with the keyword. Which makes it hardly encapsulated; since the caller must supply the array, it is obligated to pass a shared array and can't just generically pass a regular array and let the processing function worry about parallelism. No, the calling function must worry about parallelism too, so that the processing function receives a shared array and needn't reallocate the array into shared space.
There is also a very strange bug, that when I dynamically allocate shared(creal)[] at certain spots, it simply hangs at allocation. Seems very random and can't find the culprit...
In the test example this works, but not in my project... This turned out to be a bug in DMD / OptLink.
EDIT2: I never mentioned, but it's for implementing the FFT (Fast Fourier Theorem). So I have no power over selecting precise cache aligned slices. All I know is the elements are of type creal and the number of elements is a power of 2 (per row / column).
you can use the std.parallelism module
T[] buff;
foreach(ref elem;parallel(buff))elem=0;
but if you want to reinvent this you can just use shared (it is thread safe to only let 1 thread accesses a certain element at the time and if you enforce this with the appropriate join() or Task.*force() so much the better)
I need a way to tag a pointer as being either part of set x or part of set y (ie: the tag has only 2 'states'), I'm that means one can assume untagged = x and tagged = y.
Currently I'm looking at using bitwise xor to do this:
ptr ^ magic = encoded_ptr
encoded_ptr ^ magic = ptr
but I'm stumped at how to determine if the pointer is tagged in the first place.
I'm using this to mark what pools nodes in a linked list come from, so that when the are delinked, they can go back to the correct perants.
Update
Just to make it clear to all those people suggesting to store the flag in extra data members, I'm limited to sizeof(void*), so I can't add new members, else I would have. Also the pools aren't contiguous, they consist of many pages, tracking the ranges would add too much overhead (I'm after a fast & simple solution, if one can call it that).
Most solution will be platform specific. here a few of them:
1) A pointer returned by malloc or new will be aligned (4, 8, 16, 32 bytes, you name it). So, on most architectures, several LSB bits of the address will be always 0.
2) And a Win32 specific way: unless your program uses 3GB switch, values of all usermode pointers are less than 0x80000000, so the highest bit can be used as flag. As bonus, it will also crash when the flagged pointer is dereferenced without being repaired.
There is no safe and portable way to make that sort of thing work. You might be able to find some system-specific bits that are always a known value (say, the most significant n bits), but that's an extremely fragile and dangerous thing to rely on. You can't tell whether a pointer is "marked" or not unless some of the bits in the pointer have known values in the first place.
A much better way to do this is to store an identifier in the structure the pointer points to.
Surely if you only have two pools, when you allocate memory for each pool you know the possible address range - so why not check whether your given pointer occurs in one or the other address range with simple pointer arithmetic?
If performance is not a big issue, two std::set's can be used.
If it's important to get this information quickly, and it's acceptable to use only 2-byte aligned pointers, the lowest bit can be used to store this information. But having "hacked" pointers may appear to be quite error-prone...
You might have ptr1 ^ magic = ptr2 with ptr1 in set X and ptr2 in set Y (unless you prove otherwise). Since (I guess) you don't have control on the pointers addresses you are given, your technique seems to be inadequate.
An alternative to Vinay solution is to store the tags as bits of a pre-allocated buffer (specially easy if the size of the list is bounded since you don't have to grow or shrink the buffer). This is a very compact and efficient solution that does not require to modify the pointed data structure.
Cheers,
-stan
How would you go about determining whether a position is already occupied or not? When the memory is allocated, all that there is in it is garbage (in C++, which is what I'm using atm). I was thinking of using an auxiliary array of bools to know whether the position is occupied, but that would demand quite a lot of additional memory.
I could also set a value for every position, but then I wouldn't be able to use said value. In both cases, I would also lose some performance initializing the values (the bools to false, the other values to 0 to indicate the position is free, for example).
Any other solutions?
Usually, you use a special placeholder element to indicate empty values. In the simplest case, this could be a null pointer but that would of course mean that you introduce an indirection; you can't store your values directly. In all other cases you would have to make allowance for the type actually stored. For example, if you stored 32 bit integers, you would have to reserve at least one predefined value (e.g. 0) as a sentinel element, thus reducing the range of values that may be stored in your hash table.
An additional array with flags is quite a good solution. Consider that this array could be reduced by a factor of at least 8 by storing bit flags instead of whole-byte variables (or even bools, which would require 4 bytes each on most architectures).
You could use boost::optional for this, instead of a raw value. That's the reason it was created, to add a not-initialized value to an item. It has a performance hit similar to initializing the values in the first place, but requires only a small amount of extra memory per item.