Efficient algorithm for iterative ranking of a vector - fortran

Suppose I've a scrambled vector of consecutive integers 1:n, say {3,6,2,1,4,5}. My problem is to find, for each element, the number of elements to its left that are smaller than itself. So I'd like the program to return {0,1,0,0,3,4} for this example. This is what I've written in Fortran:
subroutine iterrank(n,invec,outvec,tempvec)
implicit none
integer :: n, i, currank
integer, dimension(n) :: invec, outvec, tempvec
tempvec = 0
outvec = 0
do i = 1,n
currank = invec(i)
outvec(i) = tempvec(currank)
tempvec(currank:n) = tempvec(currank:n) + 1
end do
return
end subroutine
It takes a temporary array (vector), and for each digit d the loop comes across, it adds 1 to every element beyond position d in the temporary vector. The next iteration then takes the appropriate element in the temporary vector as the count of elements smaller than itself. My questions are:
1) I believe this is of complexity O(n^2), since there are O(n) writes to the temporary vector in each iteration of the loop. Am I correct?
2) Is there a more efficient way of doing this for large n (say, >100k)?

I believe this would be more efficient, and you could also reduce the temporary integer array to a single byte.
subroutine iterrank(n,invec,outvec,tempvec)
implicit none
integer :: n, i, currank
integer, dimension(n) :: invec, outvec, tempvec
tempvec = 0
!outvec = 0 ! no need to initialize something overwritten below
do i = 1 , n
currank = invec(i)
outvec(i) = sum( tempvec(1:currank) )
tempvec(currank) = 1
end do
end subroutine
The gain is that you are only writing twice per index, however you are reading elements a maximum of n*n times.
EDIT:
I haven't tried this, but it should do less reads, with a possible overhead of branching. It is possibly faster for extremely large arrays, I would however expect it to be slower for short arrays:
subroutine iterrank(n,invec,outvec,tempvec)
implicit none
integer :: n, i, currank, prevrank
integer, dimension(n) :: invec, outvec, tempvec
tempvec = 0
outvec(1) = 0
tempvec(invec(1)) = 1
do i = 2 , n
prevrank = invec(i-1)
currank = invec(i)
if ( abs(prevrank-currank) > currank ) then
outvec(i) = sum( tempvec(1:currank) )
else if ( prevrank < currank ) then
outvec(i) = outvec(i-1) + sum( tempvec(prevrank:currank) )
else
outvec(i) = outvec(i-1) - sum( tempvec(currank:prevrank-1) )
end if
tempvec(currank) = 1
end do
end subroutine iterrank

Complete rewrite of the answer. If the memory is not a concern, you can add another vector and use an algorithm like the one bellow. The additional vector is used to compute the permutation. Thanks to the fact that the original vector is a permutation of integer 1 to n, the permutation is computed in O(n). with vectors of size 100k on my computer, this algorithm runs in 1.9 sec in average (100 runs) and the initial proposition of zeroth is 2.8 sec in average. I suggested this solution simply because zeroth said he did not test his new solution, you will test and use the best one.
subroutine iterrank(n,invec,outvec,tempvec,ord)
implicit none
!
integer :: n, i, currPos, prevPos, currOut, prevOut
integer, dimension(n) :: invec, outvec, tempvec,ord
!
tempvec = 0
do i = 1, n
ord(invec(i)) = i
end do
!
currPos = ord(1)
tempvec(currPos) = 1
currOut = 0
outvec(currPos) = currOut
! last = 0
do i = 2 , n
prevPos = currPos
currPos = ord(i)
!
if(currPos>prevPos)then
currOut = currOut+sum( tempvec(prevPos:currPos) )
else
currOut = sum( tempvec(1:currPos) )
end if
!
outvec(currPos) = currOut
tempvec(currPos) = 1
end do
!
end subroutine iterrank
The down side of this solution is the random access to vectors outvec and tempvec, that does not make the best use of cache and registers. It is possible to solve that and reduce significantly the time, possibly at the expense of additional temporary vectors.

Related

Inverse mapping in Fortran

Suppose I have an array A in Fortran of dimension 10 with numbers.
However I'm only interested in a subset of those numbers (for example 3).
I store those number in a smaller array B
B(1) = A(1)
B(2) = A(5)
B(3) = A(6)
I can also define a mapping table to store index 1, 5, 6 for example
MAP(1) = 1
MAP(2) = 5
MAP(3) = 6
How can I create an inverse map INVMAP such that
INVMAP(1) = 1
INVMAP(5) = 2
INVMAP(6) = 3
with the constrain that INVMAP has dimension 3 (and not 10).
The point is that the array A is too big to be stored in memory and B
is obtained iteratively (A is never really allocated).
Considerations:
I do not care about the 7 discarded values but I care about the position of the one we keep.
Since MAP and INVMAP are storing positions, there will never be collision (its a one to one correspondence).
Maybe it could be possible with HASH or Fortran table but I'm not really sure how because I'm mapping numbers, not keys. Any idea ?
Thanks a lot,
Sam
Here's a very simple solution. No Fortran on this machine so not entirely sure that I have the syntax absolutely correct. Define a derived type like this:
type :: row
integer :: a_index
integer :: a_value ! I've assumed that your A array contains integers
! use another type if you want to
end type
then
type(row), dimension(100) :: b ! In practice you'll probably want b to be
! allocatable
and
b(1) = (1, a(1)) ! each row of b contains the value at an index into a and
! the index
b(2) = (5, a(5))
b(3) = (6, a(6))
Now your map function is simply, in pseudo-code, map(n) = b(n)%a_index
and your inverse map is, again in pseudo-code, invmap(n) = findloc(b%a_index, n).
Since the inverse map is a simple scan, it might become too time-consuming for your taste when b becomes large. Then I might introduce an auxiliary index array pointing into b at intervals, or I might go crazy and start a binary search of b%a_index.
I can also define a mapping table to store index 1, 5, 6 for example
MAP(1) = 1
MAP(2) = 5
MAP(3) = 6
I don't know if you know, but Fortran has a nice feature (one of my favorites) known as Vector Subscripts. You can pass an 'array of indices' as an index to an array, and get the elements corresponding to each index, like this:
integer :: A(10) = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
integer :: map(3) = [1, 5, 6]
print *, A(map)
! outputs 10 50 60
How can I create an inverse map INVMAP such that
INVMAP(1) = 1
INVMAP(5) = 2
INVMAP(6) = 3
Well, if your INVMAP will have a domain and an image of different sizes, it must be a function, not an array. I mean, INVMAP cannot be an array of length 3 and accept indices from 1 to 6, right? Arrays in Fortran (and in most languages) imply contiguous indices.
The intrinsic function FINDLOC can be pretty handy here (I am asuming your function is bijector).
function invmap(map, ids)
integer :: ids(:), map(:), invmap(size(ids))
invmap = [(findloc(map, i), i = 1, size(ids))]
end
You could use this function to relate each map value to its position on map
integer :: myinvmap = invmap(map, [6, 1, 5])
print *, myinvmap ! outputs 3 1 2
print *, invmap(map, [5]) ! outputs 2
The point is that the array A is too big to be stored in memory and B
is obtained iteratively (A is never really allocated).
Now, if you will never allocate the big array, then its values will also be accessed by some function (you can consider it a function actually). You have basically two options here:
Have two arrays, one with the values got from big_array_function, and one with the parameter you passed to big_array (the indices).
Have one array of pairs [index, value]. It is the answer that #HighPerformanceMark provided.
Alternatively... (and not tested...)
Integer, dimension(100) :: A
Logical, dimension(100) :: A_Mask
Integer, dimension( 3) :: B
Integer, dimension(. 3) :: A_pos
Integer, dimension(. 3) :: I, J
A_Mask = .false.
A_Mask(1) = .true.
A_Mask(1) = .true.
A_Mask(1) = .true.
B = PACK(A, MASK=A_Mask)
J = 0
Collect_Positions: Do I = 1, SIZE(A)
If(.not. A_Mask(I)) CYCLE Collect_Positions
J = J+1
A_Pos(J) = I
ENDDO Collect_Positions
...
And then if one want to UNPACK, then the mask has the position... so it is possible to not worry about the position of A in general sense (but may be needed in the OP's case)

Is there anyway to check if n number of terms in a row of a matrix are equal in fortran?

I was wondering if there was a quick way to have fortran look throughout a maxtrix’s rows and determine if n number of terms are equal.
I wasn’t able to find a question similar to mine and can’t find any help online.
Assuming we consider a matrix of integers this comes at O(N³) cost, N being the dimension of the matrix. Essentially, for each row, you need to compare each element to each other element in that row, requiring O(N³) operations. You probably need to write that yourself, but its no big deal, loop over the rows and check for each separately, if some element appears n-times
integer :: M(N, N) ! matrix to check
integer :: n, i, j, k, counter ! flag if a value appears n times
logical :: appears
appears = .false.
do i = 1, N ! loop over the rows
do j = 1, N ! loop over the entries
counter = 1
do k = j + 1, N
! check if the elements are the same, if yes, increase the counter
! exact implementation depends on type of M
if(M(i, k) == M(i, j)) counter = counter + 1
! check if this element appears n times
if(counter == n) appears = .true.
! or even more often?
if(counter > n) appears = .false.
end do
end do
end do
You can adapt that to your need, but you can do it like this.
Here's a pragmatic alternative to the solutions #RodrigoRodrigues has already provided. In the absence of any good evidence (the question is seriously underspecified) that we need to be concerned about asymptotic complexity and all that good stuff, here's a simple straightforward function which took me about 5 minutes to design, code, and test.
This function accepts a rank-1 array of integers and spits back a rank-1 array of integers, each element corresponding to the count of that element in the input array. If that description confuses you, bear with me and read the code which is fairly simple:
FUNCTION get_counts(arr) RESULT(rslt)
INTEGER, DIMENSION(:), INTENT(in) :: arr
INTEGER, DIMENSION(SIZE(arr)) :: rslt
INTEGER :: ix
DO ix = 1, SIZE(arr)
rslt(ix) = COUNT(arr(ix)==arr)
END DO
END FUNCTION get_counts
For the input array [1,1,2,3,4,1,5] it returns [3,3,1,1,1,3,1]. If OP wants to use this as the basis of a function to see if there is any value which occurs n times then OP could write
any(get_counts(rank_1_integer_array)==n)
If OP is concerned to know what elements occur n times then it is fairly straightforward to use the result of get_counts to refer back to the original array to extract that element.
This solution is pragmatic in the sense that it is parsimonious with my time rather than with the computer's time. My solution is somewhat wasteful of space, which may be an issue for very large input arrays. Any of Rodrigo's solutions may outperform mine, in both time and space, in the limit.
I was wondering if there was a quick way to have fortran look throughout a maxtrix’s rows and determine if n number of terms are equal.
As far as I understood your problem, this is what you want:
a function with the signature: (integer(:), integer) -> logical
this function receives the 1-D array line and checks if there is any value that appears at least n times in the array
the function is not supposed to indicate what or how many were those values, their positions or the exact number of repetitions
There are many ways to achieve this. "What is the most efficient?" It will depend on the specific conditions of your data, system, compiler, etc. To illustrate that, I came out with 3 different solutions. All of them give the correct answer, of course. You are advised to test each of them (or any other you come up with) with samples of your real data.
Naive solution #1 - good 'ol do loops
This is the default algorithm. It traverses line and stores each value into the aggregator list packed, that has each distinct value found so far, along with how many times they appeared. In the moment that any value reaches n repetitions, the fuction returns .true.. If no values reached n repetitions, and there is no more chance to complete the predicate, it returns .false..
I say defalut because it is the minimum linear algorith (that I figured out) based on good ol' do loops. This would probably be the best for the general case, if you have zero information about the nature of the data, system or even the programming language specifics. The aggregator is there to terminating the function as soon as the condition is met, but at the cost of an aditional list-traverse (on its length). If there are many different values in the data and n is large, the aggregator gets too long and the look-up can become an expensive operation. Also, there is almost no room for parallelism, vectorization and other optimizations.
! generic approach, with loops and aggregator
pure logical function has_at_least_n_repeated(line, n)
integer, intent(in) :: line(:), n
integer :: i, j, max_repetitions, qty_distincts
! packed(1,:) -> the distinct integers found so far
! packed(2,:) -> number of repetitions of each distinct integer so far
integer :: packed(2, size(line) - n + 2)
if(n < 1 .or. size(line) == 0) then
has_at_least_n_repeated = .false.
else if(n == 1) then
has_at_least_n_repeated = .true.
else
packed(:, 1) = [line(1), 1]
qty_distincts = 1
max_repetitions = 1
i = 1
! iterate until there aren't enough elements left to reach n repetitions
outer: do, while(i - max_repetitions <= size(line) - n)
i = i + 1
! test for a match on packed
do j = 1, qty_distincts
if(packed(1, j) == line(i)) then
packed(2, j) = packed(2, j) + 1
if(packed(2, j) == n) then
has_at_least_n_repeated = .true.
return
end if
max_repetitions = max(max_repetitions, packed(2, j))
cycle outer
end if
end do
! add to packed
qty_distincts = qty_distincts + 1
packed(:, qty_distincts) = [line(i), 1]
end do outer
has_at_least_n_repeated = .false.
end if
end
Naive solution #2 - trying for some vectorization
This approach tries to take advantage of the arraysh-nature of Fortran and the fast implementations of the intrinsic functions. Instead of an internal do loop, there is a call to the intrinsic count with an array argument, allowing the compiler to do some vectorization. Also, if you hane any tool for parallelism or if you know how to work with coarrays (and your compiler supports), you could use this approach to implement them.
The disadvantage here is that the function does a scan for all elements, even if they appeared before. So, this is more suitable when there are many different posible values in your data, with few repetitions. Although, it would also be easy to add a cached list with the past values, and use the intrinsic any, passing the cache as a whole array.
! alternative approach, intrinsic functions without cache
pure logical function has_at_least_n_repeated(line, n)
integer, intent(in) :: line(:), n
Integer :: i
if(n < 1 .or. size(line) == 0) then
has_at_least_n_repeated = .false.
else if(n == 1) then
has_at_least_n_repeated = .true.
else
! iterate until there aren't enough elements left to reach n repetitions
do i = 1, size(line) - n + 1
if(count(line(i + 1:) == line(i)) + 1 >= n) then
has_at_least_n_repeated = .true.
return
end if
end do
has_at_least_n_repeated = .false.
end if
end
Naive solution #3 - functional style
This is my favorite (personal criteria). I like functional languages and I enjoy borrowing some aspects of it into imperative languages. This approach delegates the calculation to an internal auxiliary recursive function. There are no do loops here. On each function call, just a section of line is passed over as argument: a shorter array with only values not checked so far. No need for cache either.
To be honest, Fortran's support for recursion is far from great - there is no tail recursion, compilers usually implement low call-stack limit, and many auto-optimizations are prevented by recursion. Even though, the algorithm is smart, I love how it looks like and I wouldn't discard it before doing some tests and comparissons.
Note: Fortran does not allow nested procedures in the contains part of a main program. For it to work as presented, you'd need to put the function in a module, submodule or make it an external function. Other option would be extracting the nested function and making it a normal function in the same scope.
! functional approach, auxiliar recursive function and no loops
pure logical function has_at_least_n_repeated(line, n)
integer, intent(in) :: line(:), n
if(n < 1 .or. size(line) == 0) then
has_at_least_n_repeated = .false.
else if(n == 1) then
has_at_least_n_repeated = .true.
else
has_at_least_n_repeated = aux(line)
end if
contains
! on each iteration removes all entries of an element from array
pure recursive function aux(section) result(out)
integer, intent(in) :: section(:)
logical :: out, mask(size(section))
integer :: left
mask = section /= section(1)
left = count(mask)
if(size(section) - left >= n) then
out = .true.
else if(n > left) then
out = .false.
else
out = aux(pack(section, mask))
end if
end
end
Conclusion
Do the tests before choosing a path to follow! I talked a litte here about my personal feeling on each approach and its implications, but it would be really nice if some of the Fortran Gurus on this site join the discussion and provide accurate information an critic.
I took the question to mean that it was to be determined whether any value in a row was repeated at least n times. To figure this out I chose to sort a copy of each row using qsort from the C standard library and then it's easy to find the lengths of each run of values.
module sortrow
use ISO_C_BINDING
implicit none
interface
subroutine qsort(base, num, size, compar) bind(C,name='qsort')
import
implicit none
integer base(*)
integer(C_SIZE_T), value :: num, size
procedure(icompar) compar
end subroutine qsort
end interface
contains
function icompar(p1, p2) bind(C)
integer(C_INT) icompar
integer p1, p2
select case(p1-p2)
case(:-1)
icompar = -1
case(0)
icompar = 0
case(1:)
icompar = 1
end select
end function icompar
end module sortrow
program main
use sortrow
implicit none
integer, parameter :: M = 3, N = 10
integer i, j
integer array(M,N)
real harvest
integer, allocatable :: row(:)
integer current, maxMatch
call random_seed
do i = 1, M
do j = 1, N
call random_number(harvest)
array(i,j) = harvest*3
end do
end do
do i = 1, M
row = array(i,:)
call qsort(row, int(N,C_SIZE_T), C_SIZEOF(array(1,1)), icompar)
maxMatch = 0
current = 1
do j = 2, N
if(row(j) == row(j-1)) then
current = current+1
else
current = 1
end if
maxMatch = max(maxMatch,current)
end do
write(*,'(*(g0:1x))') array(i,:),'maxMatch =',maxMatch
end do
end program main
Sample run:
0 0 0 2 0 2 1 1 1 0 maxMatch = 5
2 1 2 1 0 1 2 1 2 0 maxMatch = 4
0 0 2 2 2 2 2 0 1 1 maxMatch = 5

Sparse Storage in Fortran: Only Reading and Writing

I have an array with multiple dimensions (the goal is to allow for about 100) and each dimension has a size of about 2^10 and I only need to store in it about 1000 double precision coefficients. I don't need to do any operation with this array aside from reading and writing into it. The code is written in Fortran 90.
I assume that if I a library like one of the ones mentioned in this answer I would be able to store the do this, but would this be optimized for the simple reading and writing operations? Is there a library that would be most efficient for that purpose?
Edit: By "simple reading and writing operations" I mean the following. Suppose
REAL(8), DIMENSION(1000) :: coeff1
INTEGER, DIMENSION(1000,5) :: index
I want to define coeff2 to store the values in coeff1 and then read itat the indices in index, that is
DO i = 1,1000
index(i,:) = [something]
coeff1(i) = [another something]
coeff2(index(i,1),index(i,2),index(i,3),index(i,4),index(i,5)) = coeff1(i)
ENDDO
Then, for any i I would like to access the value of
coeff2(index(i,1),index(i,2),index(i,3),index(i,4),index(i,5))
as quickly as possible. Being able to do this fast is what I mean by "efficient".
Since the indices in [something] are at most 2^10 I am currently defining coeff2 as follows:
REAL(8), DIMENSION(2**10,2**10,2**10,2**10,2**10) :: coeff2
but this is too wasteful of memory specially since I need to increase the number of dimensions, now 5, to the order of 100 and most elements of this array are equal to 0. So, another measure of efficiency that is relevant to me is that the memory necessary to store coeff2 should not explode as I increase the number of dimensions.
Well, It's still not totally clear to me the nature of your data and the way you want to use it.
If what you need is indexed data, whose indices are not consecutive,
Sparse matrix can be an answer, and there are many solutions already implemented over the internet (as shown in the link you provided). But maybe it would be overkill for what I think you are trying to do. Maybe a simple datatype could serve your purpose, like this:
program indexed_values
implicit none
type :: indexed
integer :: index
real(8) :: value
end type
integer, parameter :: n_coeffs = 1000
integer, parameter :: n_indices = 5
integer :: i
real(8), dimension(n_coeffs) :: coeff1
integer, dimension(n_coeffs, n_indices) :: index
type(indexed), dimension(n_coeffs, n_indices) :: coeff2
type(indexed) :: var
do i = 1, n_coeffs
index(i, :) = [1, 2, 4, 16, 32] * i ! your calc here
coeff1(i) = real(i * 3, 8) ! more calc here
coeff2(i, :)%index = index(i, :)
coeff2(i, :)%value = coeff1(i)
end do
! that's how you fetch the indices and values by stored position
var = coeff2(500, 2)
print*, var%index, var%value ! outputs: 1000 1500.0
! that's how you fetch a value by its index
print*, fetch_by_index(coeff2(500, :), 1000) ! outputs: 1500.0
contains
real(8) function fetch_by_index(indexed_pairs, index)
type(indexed), dimension(:) :: indexed_pairs
integer, intent(in) :: index
integer :: i
do i=1, size(indexed_pairs)
if(index == indexed_pairs(i)%index) then
fetch_by_index = indexed_pairs(i)%value
return
end if
end do
stop "No value stored for this index"
end
end
The provided function for fetching values by its indices could be improved if your indices will be alwyas stored in ascending order (no need to traverse the whole list to fail). Moreover, if you will assing a constant result of coeff1 to all the indices at each row, you could do even better and just not having a coeff2 array at all, just have coeff1 for values and index for the indices, and correlate them by position.

Filling an array with unknown size in fortran90

I want to fill an array with unknown size in fortran90.
This is the equivalent code in MATLAB :
for i=1:10
A[i] = i
end
I know that i can pass the size, but
How can i do this in fortran90 without passing the size of the array. I read that we can use pointers, but i really dont know how to deal with pointers
I understand you want to start adding elements to an array before you know the final size of the array.
As an example, you want to read values from a file until you reach the end of the file, not knowing how many values there are.
There are three ways I can think of:
Create an array of sufficient size, and remember the final value.
integer :: a(200), n
n = 1
do
a(n) = <value>
if (<finished>) exit
n = n + 1
end do
<use a(1:n)>
Create two allocatable arrays, when you reach the end of one, make the other bigger, and swap them:
integer, allocatable :: a(:), tmp(:)
integer :: i, n
n = 8
allocate(a(n))
i = 1
do
if (i > n) then
allocate(tmp(2*n))
tmp(1:n) = a(:)
call move_alloc(tmp, a)
n = n * 2
end if
a(i) = <value>
if (<finished>) exit
i = i + 1
end do
allocate(tmp(i))
tmp(:) = a(1:i)
call move_alloc(tmp, a)
I do no longer recommend this. Pointers can be confusing and create weird and hard-to-debug bugs. But I leave it in for posterity: Create a linked list (here using a stack)
type t_node
integer :: value
type(t_node), pointer :: next => NULL()
end type t_node
type(t_node), pointer :: list, tmp
integer, allocatable :: a(:), i, n
nullify(list)
nullify(tmp)
do
allocate(tmp)
tmp % value = <value>
tmp % next => list
list => tmp
nullify(tmp)
if (<finished>) exit
n = n + 1
end do
allocate(a(n))
do i = n, 1, -1
a(i) = list % value
tmp => list
list => list % next
deallocate(tmp)
end do
The way I read your question, you have a subroutine that needs to fill an array, but that array is of unknown size, and you do not want to pass in the size. So you do NOT want this:
SUBROUTINE FILL( A, N )
INTEGER N
INTEGER A(N)
INTEGER I
DO I=1,N
A(I) = I
END DO
END SUBROUTINE FILL
Instead, you want to get the SIZE of the array:
SUBROUTINE FILL( A )
INTEGER A(:)
INTEGER I
DO I=1,SIZE(A)
A(I) = I
END DO
END SUBROUTINE FILL

how to tell the compiler that a loop can be safely parallelized?

I'm updating a few elements in a large array.
The updates consists of:
multiplying the current value by ten (if it's not zero)
clearing the current value
moving the updated value to a new position in the array
I know there will be no collision when a move occurs.
How can I tell the compiler that it can safely parallelized the loop?
do i = 1, 1e6
if ( v[i] /= 0 ) then
temp = v[i] * 10
v[i] = 0
ndx = get_move_to_ndx(i)
v[ndx] = temp
end if
end do
I'm on ifort, but I guess this is compiler independent.
Here is a mongrel approach, so you have some ideas using temporary vectors. The WHERE may not be correct, you would have to try it. The Main advantage of WHERE/ELSEWHERE is readability, as it is usually not as fast as loops... Just easier to read.
!DIR$ SIMD
FillTemp: Do I = 1, 1000000
Temps(I) = v(I)*10
ENDDO FillTemp
!$OMP PARALLEL DO
FindIndex: Do I = 1, 1000000
ndx_vect(I) = get_move_to_ndx(i)
ENDDO FindIndex
WHERE( Temps /= 0 )
V = 0
ELSEWHERE
v(ndx_Vect) = tempz
ENDWHERE