I have a few questions to better understand the problem.

1. On how many nodes do you run the calculation, how many CPU cores are available on each node?
2. How many OpenMP thredas do you use?
3. There is a file juDFT_times.json created. Could you attach that?

Best, Gregor.

The diagonalization seems to consume a larger fraction of the runtime than what I would expect. This may or may not be acceptable. But it should be understood. So two more questions:

1. Which library do you use for the diagonalization?
2. What is the output of "grep nvd out.xml"

Here is what I can say:

1. nvd is about the number of LAPW basis functions (without LOs) that you have in your system. 23.000 LAPW basis functions really is a lot for 82 atoms (typical sizes are about 100/atom). This seems to be due to a very high choice for Kmax. Was this automatically generated or the result of a convergence study? It may be reasonable because you have rather small bond lengths in your system and thus also rather small MT spheres.

2. It is good that you mix MPI parallelization with OpenMP parallelization. Maybe a little bit less OpenMP would improve the CPU utilization. In that plot you can see that at the moment we would expect about a factor 5 speedup when 12 OpenMP threads are used, but with 6 OpenMP threads you also nearly get this speedup. I typically use something like 4 to 8 OpenMP threads. If you can rebalance the different parallelization schemes in that direction you might benefit. You can also use more than 32 MPI processes. 64 or 128 may be reasonable. You can also see in the plot that the diagonalization actually is the limiting code part when it comes to the OpenMP parallelization. In your system you have many LAPW basis functions for not so many atoms. This leads to a large runtime demand for the diagonalization and thus this becomes the determining bottleneck.

3. Since the CPU utilization is strongly determined by the diagonalization, which is processed in a separate library there are only a few measures you can take to improve this situation. Rebalancing the different parallelization schemes is one approach, another would be the use of a more efficient diagonalization library. It may be a good idea to try out the ELPA library. This may give you some boost.

Another point: At the moment I am doing convergence studies on numerous Oxide materials, also to prepare a more efficient default parameter setup... With respect to convergence it may be a good idea to increase the Oxygen MT radius. In my studies I find that this radius should be at least 1.3 Bohr radii. Zinc probably can be slightly smaller (you have to make compromises on the other atoms). I don't know what demands Hydrogen has.