I am trying to mix fortran with octave for some of my works.
I created a matrix in fortran and wrote that in a datafile with the following command.
do i=1,n
write(1,*) (/(a(i,j),j=1,n)/)
end do
Then I am loading this matrix in octave using dlmread and diagonalising the matrix using eigs there. Then again the array containing the eigenvalues is saved in a file in octave.
After that I am loading the file in fortran and reading that array.
The drawback of this method is, it takes a huge time to write the matrix and read, and also it takes space like 2gb for a 10k*10k matrix. Is there any other way of calling the octave routines like eigs in fortran to make it faster?
Related
I am currently "optimizing" a scientific modelling program developed in Fortran 95. This program is basically making heavy computations in 3D to solve some equations, in addition numerous variable have to be saved and used ~ 50 tables with sizes likes (50; 50; 10000), I even have some 5D tables with sizes like (6;6;15;15;10000) to save in order to reduce the computation time.
I developed a perfectly working version of this code using a python3 interface to control my runs. Basically python is calling a fortran module containing my code to obtain all the results from my modelling. The problem with this method is that I cannot parallelize my code in some time consuming regions. Moreover, I would benefit from the computational time advantage of Fortran for a post treatment of the models that is now partially done in python due to interface.
In the first part of my optimization campaign for this code I want to add a control of the runs with Fortran. A program would call the module containing my code to obtain all the necessary and heavy variables. The Python interface would still be presented, the switch between the Fortran and python control run being done in the compilation in the Makefile directly, this Makefile is already done, everything is compiling well and the python interface is still perfectly working.
My troubles are concerning the Fortran control program and its gestion of the allocated memory I assume. As the size of my tables are not known in advance and requires to open some files I have to declare all my variable as ALLOCATABLE. I then allocate them with the correct sizes before calling my module containing my code. When calling my code errors related to memory problems are appearing, with the error message "Program received signal SIGSEV: Segmentation fault - invalid memory reference". This error appears when I'm setting a table to 0d0, if I'm reducing the size/precision of my modelling the program can proceed a bit further before crashing hence the memory related problem. I think that I'm doing something not correct in the utilisation of the variables between my control and my modelling module. Maybe some variables are stored in the wrong memory space, I precise that I'm using gfortran on ubuntu 22.04.1.
I have different possibilities to try to solve this issue using derived types and pointers or simply by breaking my modelling module. Before going into these heavy structural modifications I wanted to know if someone has experience an equivalent problem and what were the solutions.
Here is a schema of the structure of my code:
Run program:
program run_model
use coordinates
use file
use mathematical
use modelling_module
implicit none
integer :: n_x, n_y, n_z
real(8),dimension(:), ALLOCATABLE:: x,y,z
+ all other output variables in 3D
.
.
.
Some operations and file opening
ALLOCATE(x(n_x),y(n_y),z(n_z))
+ all other variables
CALL modelling(n_x, n_y, n_z, output variables)
end program run_model
Modelling module in a separated file:
module modelling_module
use coordinates
use file
use mathematical
implicit none
private
public :: modelling
contains
subroutine modelling(n_x, n_y, n_z, output variables)
integer, intent(in):: n_x, n_y, n_z,
real(8),dimension(n_x), intent(out):: x
real(8),dimension(n_y), intent(out):: y
real(8),dimension(n_z), intent(out):: z
+ all output variables
Computation of the model
.
.
.
end subroutine modelling
end module modelling_module
Thank you in advance for your answers !
I have been using Fortran for a few months now, but I am self-taught and have only been learning it by reading someone else's codes so my knowledge of Fortran is very limited. I wrote this function which is meant to read a text file containing data and save these data in an array. Since I don't know the size of the data, I choose to allocate the array within the function.
FUNCTION RSEBIN(NAMEIN,NZNSEB)
IMPLICIT DOUBLE PRECISION (A-H, O-Z)
INTEGER DSEBTP, IIND, NZNSEB
CHARACTER(LEN=75) :: FILNAM
CHARACTER NAMEIN*(*)
REAL, ALLOCATABLE :: RSEBIN(:,:)
WRITE (FILNAM,1500) 'Extra_InputFiles/SEB_inputs/SEB_', NAMEIN,
2 '.txt' !Define the path and name of the input data text file
1500 FORMAT (A32,A,A4)
OPEN (UNIT=101, FILE=FILNAM, STATUS='OLD')
READ(101,*) !Skip the header
DSEBTP = 0
DO
READ(101,*,IOSTAT=IO) TRASH
IF (IO.NE.0) EXIT !Exit the loop when last line has been reached
DSEBTP = DSEBTP + 1 !Counts how many time periods inputs are set for the input data type
END DO
REWIND(101) !Rewind text file to read the inputs
ALLOCATE(RSEBIN(DSEBTP,NZNSEB+1)) !Allocate the input data array
READ(101,*) !Skip the header
DO 1510 ISEBTP=1,DSEBTP
READ(101,*) (RSEBIN(ISEBTP,IIND), IIND=1, NZNSEB+1) !Save the data in the main array
1510 CONTINUE
CLOSE (UNIT=101)
RETURN
END FUNCTION
I then use this function in another subroutine with this following statement:
ASEBAT = RSEBIN('AirTemperature',NZNSEB) !Allocate the air temperature array (first column is time)
When I try to run the program, I get a "Insufficient virtual memory" error. After a quick search, I discovered that this error usually occurs when one is allocating huge arrays. However, during my tests, I was only using a 3 X 5 array. After a few more tests, I realized that the function works fine if I declare the dimensions of my array RSEBIN rather than making it allocatable and allocating it in the function. However, this solution is not sustainable for me as I want this function to be able to read text files of various dimensions.
Does anyone have an idea why I have such error? Should I avoid allocating arrays in a function? As I said previously, I am fairly new to Fortran and I am pretty sure my code has many issues, so I apologize for my primitive code writing and would be grateful for any tip.
Also, I should note that I'm using the Intel Fortran Compiler from oneAPI for Windows. I recently switched from the fortran compiler in Intel XE, with which, if I can recall, I was using a similar function without any issue.
Thanks!
I have been trying to write a simple program to perform an fft on a 1D input array using fftw3. Here I am using a seismogram as an input. The output array is, however, coming out to contain only zeroes.
I know that the input is correct as I have tried doing the fft of the same input file in MATLAB as well, which gives correct results. There is no compilation error. I am using f95 to compile this, however, gfortran was also giving pretty much the same results. Here is the code that I wrote:-
program fft
use functions
implicit none
include 'fftw3.f90'
integer nl,row,col
double precision, allocatable :: data(:,:),time(:),amplitude(:)
double complex, allocatable :: out(:)
integer*8 plan
open(1,file='test-seismogram.xy')
nl=nlines(1,'test-seismogram.xy')
allocate(data(nl,2))
allocate(time(nl))
allocate(amplitude(nl))
allocate(out(nl/2+1))
do row = 1,nl
read(1,*,end=101) data(row,1),data(row,2)
amplitude(row)=data(row,2)
end do
101 close(1)
call dfftw_plan_dft_r2c_1d(plan,nl,amplitude,out,FFTW_R2HC,FFTW_PATIENT)
call dfftw_execute_dft_r2c(plan, amplitude, out)
call dfftw_destroy_plan(plan)
do row=1,(nl/2+1)
print *,out(row)
end do
deallocate(data)
deallocate(amplitude)
deallocate(time)
deallocate(out)
end program fft
The nlines() function is a function which is used to calculate the number of lines in a file, and it works correctly. It is defined in the module called functions.
This program pretty much tries to follow the example at http://www.fftw.org/fftw3_doc/Fortran-Examples.html
There might just be a very simple logical error that I am making, but I am seriously unable to figure out what is going wrong here. Any pointers would be very helpful.
This is pretty much how the whole output looks like:-
.
.
.
(0.0000000000000000,0.0000000000000000)
(0.0000000000000000,0.0000000000000000)
(0.0000000000000000,0.0000000000000000)
(0.0000000000000000,0.0000000000000000)
(0.0000000000000000,0.0000000000000000)
.
.
.
My doubt is directly regarding fftw, since there is a tag for fftw on SO, so I hope this question is not off topic
As explained in the comments first by #roygvib and #Ross, the plan subroutines overwrite the input arrays because they try the transform many times with different parameters. I will add some practical use considerations.
You claim you do care about performance. Then there are two possibilities:
You do the transform only once as you show in your code. Then there is no point to use FFTW_MEASURE. The planning subroutine is many times slower than actual plan execute subroutine. Use FFTW_ESTIMATE and it will be much faster.
FFTW_MEASURE tells FFTW to find an optimized plan by actually
computing several FFTs and measuring their execution time. Depending
on your machine, this can take some time (often a few seconds).
FFTW_MEASURE is the default planning option.
FFTW_ESTIMATE specifies that, instead of actual measurements of
different algorithms, a simple heuristic is used to pick a (probably
sub-optimal) plan quickly. With this flag, the input/output arrays are
not overwritten during planning.
http://www.fftw.org/fftw3_doc/Planner-Flags.html
You do the same transform many times for different data. Then you must do the planning only once before the first transform and than re-use the plan. Just make the plan first and only then you fill the array with the first input data. Making the plan before every transport would make the program extremely slow.
I have a 2-D double-precision array in MATLAB that contains specific data.
I want to use this array in c++, so I save the array in a mat-file.
I know that MATLAB has some c functions that provide reading mat-file in c++ (matdsgn , matOpen , ...), but I don't know how to use them in a c++ program. Actually, I don't know how to use a C library in C++.
Any help would be appreciated.
If you have MATLAB 2017a, there is a built-in function. See this MATHWORKS link: Math Works
According to this link and this one, it is said that opencv is much faster than matlab. First link is written in March 2012, second one is a bit later than that.
In the first link, it says, "Programs written in OpenCV run much faster than similar programs written in Matlab." and rates Matlab: 2/10 and OpenCV: 9/10
Consider, I have two float Matrix whose size are 1024*1024(mat1 and mat2). I want to correlate this matrices.
In matlab,
corr2(mat1,mat2); //70-75 ms
In opencv, c++
Mat result(1,1,CV_32F);
matchTemplate(mat1,mat2,result, CV_TM_CCOEFF_NORMED); // 145-150 ms
As far as I know, c and c++ are in approximately same speed.
So, I wonder, why matlab is faster than opencv/c++ while doing cross correlation. Is it because I am comparing wrong things (even though the results are same) or is the cross correlation implementation of matlab is double faster than opencv implementation?
Note that, I'm using Matlab 2013a and Visual Studio 2010.
Thanks,
Matlab built in functions comes with mkl and opencv's dont. So if two exactly equivalent functions are present in both, matlab is likely to be faster(much) than opencv. I have tried to do pseudo inverse on a large matrix and matlab beat everything(openblas,Armadillo,self integrated mkl etc) by at least 2 times. Then I just stopped figuring out why and just load the data into matlab and let it do the thing. opencv is by far the slowest. Try matrix multiplication on a 10000x10000 matrix in opencv. it took 10 minutes on my laptop. Matlab took 1 minute.
Matlab is not as bad as you may think at doing matrix calculations. For many of the Basic Linear Algebra operation Matlab is calling rutines written in fortran and c++. So as long as you dont use loops and formulate it in matrix operations Matlab is actually very fast.
http://www.mathworks.se/company/newsletters/articles/matlab-incorporates-lapack.html
In your scenario, there is no reason to expect matlab to be slower. You are calling a single function, the overhead caused by the language interpreter and passing the data to a native function (mex function) has to be paid only once.
If you would call the same function 1024 times for a small 32*32 matrices, you will probably notice the overhead (unless the JIT-Compiler finds a neat trick to optimize the code).
Matlab can be fast if you vectorize everything and use native functions.
But if you would do some operations in a loop i.e.
A = zeros(100,100);
for m = 1:100
for n = 1:100
A(m, n) = 1/(m + n - 1);
end
end
vs.
Mat A(100, 100, CV_64F);
for (int r = 0; r < A.rows; r++)
for (int c = 0; c < A.cols; c++)
A.at<double>(r, c) = 1 / (r + c - 1);
you would notice the difference.
For correlation functions (and many more) matlab uses an advance libraries which uses an advanced instruction set.
However Matlab is smart than you think, Matlab Checks on runtime if the operation would execute faster on spatial domain or frequency domain, than execute fastest solution.
I couldn't find a mention for corr2, however I found for normxcorr2
Calculate cross-correlation in the spatial or the frequency domain, depending on size of images.