How do dmapped domains in chapel language get actually mapped onto? - chapel

Question

I need to know few things about array element allocation over domain map in chapel
Let me keep this as short as possible
region = {1..10,5..10}
regionbox = {1..5,1..5}
grid2d = /*a 2D arrangement of locales*/
Space = domain(2) dmapped Block( boundingBox = regionbox,
target_locales = grid2d
) = region.
var : myarray[Space] int;
Now Space is a distributed domain.
So here comes in.
In a distributed domain, whether we have to keep all our indexes in each locality that is
For the above example.
whether we have to keep the indexes which maps to locales, locally on all locales ?
I hope that domain map supports global-view programming so when we are accessing myarray[3,5], it dynamically maps to associative locale using the dist.
Please correct me If I'm wrong
And how are arrays allocated over the distributed domains?
Is it that domain maps have some features, which calculate the individual local size at start, from the given parameters, and allocate local_size elements in each locale ?
Like
blocking 10 elements over 2 locales needs a local size of 5.
I want to know how the array elements are created over the distributed domain and also whether the index which are mapped to locality according to distribution, got stored in that locality ?
Please let me know if this question needs more info
Thank you for your kind help

Answer 1

As with your previous question, the answer to this question depends on the specific domain map. The Chapel language and compiler expect a domain map (and its implementation of domains and arrays) to support a standard interface, but how it implements that interface is completely up to its author. This interface includes things like "allocate a new domain for me", "allocate a new array over that domain for me", "iterate over the indices/elements of the domain/array", "randomly access the array", etc. Thus, a given domain map implementation may be very space efficient and minimal, or it can allocate everything on every locale redundantly, as its author thinks best.
That said, if we consider standard domain maps like Block, they behave the way you would expect: E.g., for a {1..n, 1..n} array mapped across 4 locales, each locale will store ~(n**2 / 4) elements of the array rather than all n**2 elements. A random access to that array will be implemented by determining which locale owns the element and having the compiler/runtime manage the communication required to get at that remote element (as implemented by the domain map). Information is stored redundantly when it only requires O(1) storage, since this redundancy is better than communicating to get the values. E.g., each locale would store the {1..n, 1..n} bounds of the domain/array since it is cheaper to store those bounds than to communicate with some centralized location to get them.
This is one of those cases where a picture can be worth a thousand words. Taking a look at the slides for the talks where we introduced these concepts (like slide 34 of this presentation) could be much more instructive than the following text-based description.
Walking through your declarations and cleaning them up a bit, here's roughly what happens as this code is executed:
const region = {1..10,5..10},
regionbox = {1..5,1..5},
grid2d = /*a 2D arrangement of locales*/;
Nothing about these declarations refer to other locales (no on-clauses, no dmapped clauses), so these would all result in domains and arrays that are stored locally on the locale where the task encountering the declarations is executing (locale #0 at the program's start-up time).
const Space : domain(2) dmapped Block( boundingBox = regionbox,
target_locales = grid2d
) = region.
The dmapped Block(...) clause causes an instance of the Block domain map class to be allocated on each locale in grid2d. Each instance of the class stores the bounding box (regionbox) and the set of target locales. Each locale also gets an instance of a class representing the local view of the distribution named LocBlock which stores the subset of the 2D plane which is owned by that locale as defined by the bounding box and the target locale set.
The declaration and initialization of Space invokes a method on the current locale's copy of the Block domain map object created in the previous step, asking it to create a new domain. This causes each locale in grid2d to allocate a pair of classes corresponding to the global and local views of the domain, respectively. The global descriptor describes the domain's indices as a whole (e.g., region) while the local descriptor describes that locale's personal subset of region.
var myarray: [Space] int;
This declaration asks the current locale's copy of the global Space domain class created in the previous step to create a new array. This causes each locale in grid2d to allocate a pair of classes representing the global and local views of the array. The global view of the array tends not to store much state and is used primarily to dispatch methods on the array to the appropriate local descriptor. The local descriptor stores the array elements corresponding to the locale's subarray.
I hope this helps clarify the issues you are asking about.