I'm trying compile a large model using ifort and with the -qopenmp flag:
FC = mpif90 and FCFLAGS = -g -m64 -qopenmp -O3 -xHost -fp-model \
precise -convert big_endian -traceback -r8
FCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT
LFLAGS = -qopenmp
CC = mpicc
CFLAGS = -g -O3
CCDEFS = BLAS LITTLE LINUX INTEGER_IS_INT _ABI64
OMP_NUM_THREADS=2
OMP_STACKSIZE=1000M
OMP_SCHEDULE=STATIC
ulimit -s unlimited
mpprun -n 192 master.exe -e "exp1" -f d1 -t 2700
However, when I try and run the model I get:
mpprun info: Starting impi run on 12 node ( 192 rank X 1 th ) for 22
==================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 19619 RUNNING AT n457
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault
(signal 11)
mpprun info: Job terminated with error
Now the thing is, if I compile this model without the OpenMP flag and run it with TotalView, there are no errors and the model executes without error.
I'm trying to find a way to track down what is going wrong. Does anyone have any ideas? Where do I start? how can I do simple tests to see why OpenMP exited with a segmentation fault?
Appreciate the help
Related
Have been tring to build this bootloader for a few days now.
https://github.com/ahtn/kp_boot_32u4
Windows pc, under mingG, WINavr and Python installed.
Im stuck at creating the elf file...
Linking: build/default-atmega32u4/kp_boot_32u4-1024-default-atmega32u4.elf
avr-gcc -mmcu=atmega32u4 -I. -Isrc -Wno-error=unused-function -DBOOT_SIZE=BOOT_SIZE_10 -DCHIP_ID=CHIP_ID_ATmega32U4 -g -DUSB_VID=0x1209 -DUSB_PID=0xBB05 -DF_CPU=16000000UL -DF_USB=UL -DBOARD=BOARD_default -DARCH=ARCH_AVR8 -D ____ -Os -fno-strict-aliasing -Wstrict-prototypes -Wa,-adhlns=build/default-atmega32u4/kp_boot_32u4-1024-default-atmega32u4.elf -std=gnu99 -Wall -Werror -MMD -MP -MF build/default-atmega32u4/obj/.dep/kp_boot_32u4-1024-default-atmega32u4.elf.d build/default-atmega32u4/obj/usb/device_descriptors.o build/default-atmega32u4/obj/main.o build/default-atmega32u4/obj/usb.o build/default-atmega32u4/obj/spm.o --output build/default-atmega32u4/kp_boot_32u4-1024-default-atmega32u4.elf -T ./ld_scripts/avr5.xn -Wl,--section-start=.text=0x7c00 -Wl,--section-start=.boot_extra=0x7ff0 -Wl,--undefined=.boot_extra -Wl,-Map=build/default-atmega32u4/kp_boot_32u4-1024-default-atmega32u4.map,--cref -Wl,--gc-sections -flto
./ld_scripts/avr5.xn:18: nonconstant expression for length
make: *** [avr-makefile/avr.mk:253: build/default-atmega32u4/kp_boot_32u4-1024-default-atmega32u4.elf] Error 1
I uploaded a palou elf to the build folder and the make process completes ok. After a bit of research i have seen many others get the error, but wondering if there is another version of avr-gcc i could use?? to be honest i am quite new to all this so may have bitten off more than i can chew (lock down mind activity)
......
**__DATA_REGION_LENGTH__ = DEFINED(__DATA_REGION_LENGTH__) ? __DATA_REGION_LENGTH__ : 0xffa0;**
.....
and causes error in memory section...
MEMORY
{
text (rx) : ORIGIN = 0, LENGTH = __TEXT_REGION_LENGTH__
data (rw!x) : ORIGIN = 0x800060, LENGTH = __DATA_REGION_LENGTH__
Thanks
M
These ** in avr5.xn is no valid linker script syntax. Fix that.
same problem here, but my mistake was putting in the wrong vales for boot section, because I could not use python.
SPM_CALL_POS = 0x8000
BOOT_SECTION_START = 0x7c00
made it work.
I am trying to run the makefile of a program in Fortran 77 that I did not write.
It specifies the following flags:
FLT_FLAGS = -native -libmil
and then uses them to compile a Fortran 77 program:
f77 -O -native -libmil -c a2.f
but fails:
f77: error: unrecognized command line option ‘-native’
makefile:107: recipe for target 'vg.a(a2.o)' failed
make: *** [vg.a(a2.o)] Error 1
The only thing I have been able to find with respect to this issue is that -native is obsolete, and maybe that's why gfortran (f77) does not recognize it: https://docs.oracle.com/cd/E37069_01/html/E37076/aevft.html#scrolltoc
You are using gfortran version 5.4. I wouldn't call f77 but gfortran directly if I were you.
The correct gfortran option similar to -native in some other compilers is -march=native or -mtune=native. See the manual and GCC: how is march different from mtune?
Don't forget to use other optimization flags like -O2 or -O3. The native option alone is not too useful.
I have reduced my program to the following example:
#include <mpi.h>
int main(int argc, char * argv[]) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
I compile and run the code, and get the following result:
My-MacBook-Pro-2:xCode_TrapSim user$ mpicxx -g -O0 -Wall barrierTest.cpp -o barrierTestExec
My-MacBook-Pro-2:xCode_TrapSim user$ mpiexec -n 2 ./barrierTestExec
================================================================================== =
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 21633 RUNNING AT My-MacBook-Pro-2.local
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault: 11 (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
If I comment out the MPI_Barrier, or run the program on only one node, the code runs fine. I am using the following compilers:
My-MacBook-Pro-2:xCode_TrapSim user$ mpiexec --version
HYDRA build details:
Version: 3.2
Release Date: Wed Nov 11 22:06:48 CST 2015
CC: clang
CXX: clang++
F77: /usr/local/bin/gfortran
F90: /usr/local/bin/gfortran
Configure options: '--disable-option-checking' '--prefix=/usr/local/Cellar/mpich/3.2_1' '--disable-dependency-tracking' '--disable-silent-rules' '--mandir=/usr/local/Cellar/mpich/3.2_1/share/man' 'CC=clang' 'CXX=clang++' 'FC=/usr/local/bin/gfortran' 'F77=/usr/local/bin/gfortran' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=-lpthread ' 'CPPFLAGS= -I/private/tmp/mpich-20160606-48824-1qsaqn8/mpich-3.2/src/mpl/include -I/private/tmp/mpich-20160606-48824-1qsaqn8/mpich-3.2/src/mpl/include -I/private/tmp/mpich-20160606-48824-1qsaqn8/mpich-3.2/src/openpa/src -I/private/tmp/mpich-20160606-48824-1qsaqn8/mpich-3.2/src/openpa/src -D_REENTRANT -I/private/tmp/mpich-20160606-48824-1qsaqn8/mpich-3.2/src/mpi/romio/include'
Process Manager: pmi
Launchers available: ssh rsh fork slurm ll lsf sge manual persist
Topology libraries available: hwloc
Resource management kernels available: user slurm ll lsf sge pbs cobalt
Checkpointing libraries available:
Demux engines available: poll select
My-MacBook-Pro-2:xCode_TrapSim user$ clang --version
Apple LLVM version 7.3.0 (clang-703.0.31)
Target: x86_64-apple-darwin15.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
This seems like it should be a trivial problem, but I can't seem to figure it out. Why would MPI_Barrier cause this simple code to seg fault?
It is supper hard to decide what is wrong with your installation. However, if you can use any of the MPI flavors, maybe you can try this one:
http://www.owsiak.org/?p=3492
All I can say, it works with Open MPI
~/opt/usr/local/bin/mpicxx -g -O0 -Wall barrierTestExec.cpp -o barrierTestExec
~/opt/usr/local/bin/mpiexec -n 2 ./barrierTestExec
and no exception in my case. It really seems to be environment specific.
I've been trying to compile and run LULESH benchmark
https://codesign.llnl.gov/lulesh.php
https://codesign.llnl.gov/lulesh/lulesh2.0.3.tgz
with gprof but I always get a segmentation fault. I updated these instructions in the Makefile:
CXXFLAGS = -g -pg -O3 -I. -Wall
LDFLAGS = -g -pg -O3
[andrestoga#n01 lulesh2.0.3]$ mpirun -np 8 ./lulesh2.0 -s 16 -p -i 10
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 30557 on node n01 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
In the webpage of gprof says the following:
If you are running the program on a system which supports shared
libraries you may run into problems with the profiling support code in
a shared library being called before that library has been fully
initialised. This is usually detected by the program encountering a
segmentation fault as soon as it is run. The solution is to link
against a static version of the library containing the profiling
support code, which for gcc users can be done via the -static' or
-static-libgcc' command line option. For example:
gcc -g -pg -static-libgcc myprog.c utils.c -o myprog
I added the -static command line option and I also got segmentation fault.
I found a pdf where they profiled LULESH by updating the Makefile by adding the command line option -pg. Although they didn't say the changes they made.
http://periscope.in.tum.de/releases/latest/pdf/PTF_Best_Practices_Guide.pdf
Page 11
Could someone help me out please?
Best,
Make sure all libraries are loaded:
openmpi (which you have done already)
gcc
You can try with parameters that will allow you to identify if the problem is your machine in terms of resources. if the machine does not support such number of processes see how many processes (MPI or not MPI) it supports by looking at the architecture topology. This will allow you to identify what is the correct amount of jobs/processes you can launch into the system.
Very quick run:
mpirun -np 1 ./lulesh2.0 -s 1 -p -i 1
Running problem size 1^3 per domain until completion
Num processors: 1
Num threads: 2
Total number of elements: 1
To run other sizes, use -s <integer>.
To run a fixed number of iterations, use -i <integer>.
To run a more or less balanced region set, use -b <integer>.
To change the relative costs of regions, use -c <integer>.
To print out progress, use -p
To write an output file for VisIt, use -v
See help (-h) for more options
cycle = 1, time = 1.000000e-02, dt=1.000000e-02
Run completed:
Problem size = 1
MPI tasks = 1
Iteration count = 1
Final Origin Energy = 4.333329e+02
Testing Plane 0 of Energy Array on rank 0:
MaxAbsDiff = 0.000000e+00
TotalAbsDiff = 0.000000e+00
MaxRelDiff = 0.000000e+00
Elapsed time = 0.00 (s)
Grind time (us/z/c) = 518 (per dom) ( 518 overall)
FOM = 1.9305019 (z/s)
I just built a cross compiler using crosstools "mips-unknown-linux-gnu-gcc" and I compiled a hello world program. The compilation went fine using the command: "mips-unknown-linux-gnu-g++ hello.cpp -o hello" but when I run the command "./hello" I get the following error:
babbage-dasnyder 50% mips-unknown-linux-gnu-g++ hello.cpp -o hello
babbage-dasnyder 51% ./hello
./hello: Exec format error. Wrong Architecture.
Why is this? Did I make the wrong cross-compiler? I'm running this on a linux machine.
Just as a note, crosstools did say it could run a trivial program:
+ /home/seas/grad/dasnyder/opt/crosstool/gcc-3.4.5-glibc-2.3.6/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-gcc -static hello.c -o mips-unknown-linux-gnu-hello-static
+ /home/seas/grad/dasnyder/opt/crosstool/gcc-3.4.5-glibc-2.3.6/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-gcc hello.c -o mips-unknown-linux-gnu-hello
+ test -x /home/seas/grad/dasnyder/opt/crosstool/gcc-3.4.5-glibc-2.3.6/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-g++
+ cat
+ /home/seas/grad/dasnyder/opt/crosstool/gcc-3.4.5-glibc-2.3.6/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-g++ -static hello2.cc -o mips-unknown-linux-gnu-hello2-static
+ /home/seas/grad/dasnyder/opt/crosstool/gcc-3.4.5-glibc-2.3.6/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-g++ hello2.cc -o mips-unknown-linux-gnu-hello2
+ echo testhello: C compiler can in fact build a trivial program.
testhello: C compiler can in fact build a trivial program.
+ test '' = 1
+ test '' = 1
+ test '' = 1
+ test 1 = ''
+ echo Done.
Done.
Just as a note, crosstools did say it could run a trivial program:
testhello: C compiler can in fact build a trivial program.
When you cross-compile to a different architecture, you are generating instructions for the new architecture and thus you may not be able to run these instructions on your current architecture. You are cross-compiling to be able to compile the code on a more powerful machine and then transfer it to the device for testing. If you are wanting to test the code directly on your machine you need to compile with your native architecture's compiler.