error with MPI parallelization

FLEUR Forum � User support � Problems while running SCF or other basic tasks � error with MPI parallelization

Topic settings

Jump to forum

Announcements New releases, new features, important bugs User support Installation Inpgen -- the FLEUR input generator Problems while running SCF or other basic tasks DOS, Bandstructures, Relaxations Magnetism, Non-collinear calculations, SOC Wannier functions Other problems FUTURE related problems General discussion Feature requests Physics with FLEUR Open discourse

#1 | error with MPI parallelization datum

Thu May 13, 2021 4:39 pm

I tried to run fleur with below command but I'm getting an error message.
mpirun -np 5 fleur_MPI

Invalid MIT-MAGIC-COOKIE-1 key Welcome to FLEUR (www,flapw,de)
MaX-Release 5.1 (www,max-centre,eu)
Running on 5 PE
1 jobs are distributed on 5 unassigned PE
Moving old out.xml to out-002.xml.
I/O warning : failed to load external entity "relax.xml"
Now copying inp_dump.xml

========== k-point set info ==========
Selected k-point list: default-1
k-point list type: mesh
3 x 3 x 3
Number of k points: 14

The OMP parallelism seems to be weird
Multiple OMPs on one core: There are 3 on cpu 8
The OMP parallelism seems to be weird
Multiple OMPs on one core: There are 2 on cpu 0
me: 0 my cpu: 0
me: 4 my cpu: 7
me: 10 my cpu: 8
me: 7 my cpu: 11
me: 2 my cpu: 6
me: 5 my cpu: 7
me: 6 my cpu: 6
me: 11 my cpu: 9
me: 1 my cpu: 10
me: 8 my cpu: 0
me: 3 my cpu: 6
me: 9 my cpu: 11
The OMP parallelism seems to be weird
Multiple OMPs on one core: There are 3 on cpu 1
me: 0 my cpu: 1
me: 8 my cpu: 7
me: 7 my cpu: 11
me: 4 my cpu: 4
me: 11 my cpu: 3
me: 3 my cpu: 2
me: 1 my cpu: 7
me: 10 my cpu: 9
me: 2 my cpu: 4
me: 9 my cpu: 8
me: 5 my cpu: 1
me: 1 my cpu: 2
me: 6 my cpu: 7
The OMP parallelism seems to be weird
Multiple OMPs on one core: There are 2 on cpu 9
me: 2 my cpu: 4
me: 7 my cpu: 11
me: 3 my cpu: 1
me: 9 my cpu: 8
me: 6 my cpu: 2
me: 8 my cpu: 0
me: 4 my cpu: 2
me: 11 my cpu: 9
me: 8 my cpu: 3
me: 10 my cpu: 8
me: 2 my cpu: 4
me: 9 my cpu: 2
me: 0 my cpu: 2
The OMP parallelism seems to be weird
Multiple OMPs on one core: There are 3 on cpu 10
me: 5 my cpu: 1
me: 3 my cpu: 3
me: 7 my cpu: 11
me: 4 my cpu: 3
me: 11 my cpu: 6
me: 4 my cpu: 7
me: 9 my cpu: 5
me: 0 my cpu: 5
me: 5 my cpu: 1
me: 5 my cpu: 10
me: 10 my cpu: 9
me: 11 my cpu: 3
me: 8 my cpu: 6
me: 0 my cpu: 9
me: 6 my cpu: 10
me: 6 my cpu: 6
me: 7 my cpu: 5
me: 10 my cpu: 8
me: 1 my cpu: 3
me: 1 my cpu: 10
me: 3 my cpu: 5
me: 2 my cpu: 5
--------------------------------------------------------
Number of MPI-tasks: 5
Number of PE/node : 5
Number of OMP-threads: 12
k-points in parallel: 1
pe's per k-point: 5
# of k-point loops: 14
--------------------------------------------------------
--------------------------------------------------------------------------
The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
Workarounds are to run on a single node, or to use a system with an RDMA
capable network such as Infiniband.
--------------------------------------------------------------------------
**************juDFT-Error*****************
Error message: MPI failed with Error_code = 53
MPI_ERR_WIN: invalid window
Error from PE:0/5
Other PEs with error messages:
1-MPI failed with Error_code = 53
MPI_ERR_WIN: invalid window
2-MPI failed with Error_code = 53
MPI_ERR_WIN: invalid window
3-MPI failed with Error_code = 53
MPI_ERR_WIN: invalid window
4-MPI failed with Error_code = 53
MPI_ERR_WIN: invalid window
*****************************************
Last known location:
Last timer:create data spaces in ei66_mpi
Timerstack:
Timer:Open file/memory for IO of eig66
Timer:Total Run
*****************************************
Rank:0 used 0.693 0.108 GB/ 726760 kB
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 819 100 80 100 739 101 939 --:--:-- --:--:-- --:--:-- 1039

Terminating all MPI processes.
Note: This is a normal procedure.
Error messages in the following lines can be ignored.

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 0.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[Me:29200] 4 more processes have sent help message help-osc-pt2pt.txt / mpi-thread-multiple-not-supported
[Me:29200] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

How can I fix the "The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release" problem and run fleur_MPI in parallel mode?

reply

#2 | RE: error with MPI parallelization datum

Fri May 14, 2021 4:42 am

Some guesses:

If you have 5 cores and are trying to run 1 MPI process per core, ensure that the environment variable OMP_NUM_THREADS has the value 1.

If you have 5 nodes each of six actual cores (12 virtual hyperthread cores) then ensure OMP_NUM_THREADS is 6.

If you have 5 nodes each of 12 actual cores then it may be that mpirun is not distributing your processes correctly across nodes.

reply

#3 | RE: error with MPI parallelization datum

Sat May 15, 2021 1:23 am (Last edited: Sat May 15, 2021 1:25 am)

"The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release." actually is a message coming from the OpenMPI library you link to Fleur. I actually don't know how this is easily fixed. You might have to get a different version of that library or link to a different MPI library. Maybe this can also be controlled by setting some environment variable. I don't know. Searching the internet reveals that you are not the only person encountering this problem with OpenMPI. Maybe some search yields a solution or workaround.

What is actually the output of "mpirun --version"?

reply

#4 | RE: error with MPI parallelization datum

Sat May 15, 2021 1:35 am

One additional comment: I saw that the error occurs in the eig66_mpi subroutine. It may be possible for you to avoid the error by going throw a different code path. An option for this is to use a number of MPI processes that equals the number of k points in your setup, so in your case 14. Just a guess, but maybe this works (for exactly this case).

reply

#5 | RE: error with MPI parallelization datum

Sat Nov 06, 2021 4:54 pm

I've been having exactly the same problem with either GCC 9.1 + OpenMPI 3.1 or Intel 19 + OpenMPI 4.0. Has anyone figured out how to fix this?

reply

#6 | RE: error with MPI parallelization datum

Mon Nov 08, 2021 2:27 pm (Last edited: Mon Nov 08, 2021 2:29 pm)

The problem seems to be that the MPI Library does not handle the one-sided communication correctly. One could try to run fleur with the '-disable_progress_thread' option which hopefully fixes this. Otherwise you could try to use the parallel hdf5 library instead of keeping the data in memory by running with '-eig66 hdf5'.

Perhaps it is also possible to persuade the MPI to allow RMA even over a "standard" (ethernet?) network. There might be installations options your system guru could known.

Hope this helps,

Daniel

reply

Convergence Problem � � output