2 A simple way to begin...
This tutorial will indicate how to perform GS calculations on hundreds processors using ABINIT.
You will learn how to use some keywords related to the “KGB” parallelization scheme where “K” stands for “k-point”, “G” refers to the wavevector of a planewave, and “B” stands for a “Band”. On the one hand, you will see how to improve the speedup of your calculations and, on the other hand, how to increase the convergence speed of a self consistent field calculation.
This lesson should take about 1.5 hour and requires access to at least a 200 CPU core parallel computer.
You are supposed to know already some basics of parallelism in ABINIT, explained in the tutorial A first introduction to ABINIT in parallel. On the contrary, the present tutorial is not a prerequisite for other tutorials about parallelism in ABINIT.
Before continuing you might work in a different subdirectory, as for the other lessons. Why not “work_paral_gspw”?
All the input files can be found in the ~abinit/tests/tutoparal/Input directory. You might have to adapt them to the path of the directory in which you have decided to perform your runs. You can compare your results with reference output files located in ~abinit/tests/tutoparal/Refs.
or the use of a specific submission file. Some scripts are given as examples in the directory ~abinit/doc/tutorial/lesson_paral_gspw/. You can adapt them to your own calculations.
Therefore, it is generally mandatory to adopt a parallelization strategy. That is to say: (i) to distribute the data in a MPI-sense on a large number of processors, and (ii) to parallelize the routines responsible for the increase of the computation time.
how to use other keywords or features related to the KGB parallelization scheme.
the publication concerning the KGB parallelisation: F. Bottin, S. Leroux, A. Knyazev, G. Zerah, Comput. Mat. Science 42, 329, (2008) “Large scale ab initio calculations based on three levels of parallelization “(available on Arxiv.org).
the presentation of F. Bottin at the ABINIT workshop 2007 (Monday 29, session 2).
You are strongly suggested to read these documents before beginning this tutorial. You might learn a lot of useful things.
However, even if you scan them with attention, you won’t learn the answer to the most frequently asked question. Why this parallelization is named KGB? We don’t know. Some people say that the reason comes from the K-point, the plane waves G and the Bands, but you can imagine everything you want.
One of the most simple way to launch the KGB parallelization in ABINIT is to add just one input variable to the sequential input file. This is paral_kgb and controls everything concerning the KGB parallelization: the use of the LOBPCG eigensolver (wfoptalg=4 or 14) of A. Knyazev, the parallel 3dim-FFT (fftalg=401) written by S. Goedecker, and some other tricks… At the time of writing, you still have to define the number or processors needed on each level of the KGB parallelization: npband, npfft and npkpt.
If ABINIT is not yet able to handle directly the number of cores and to launch from scratch an efficient distribution of processors, you can already use this parallelization as a black box. It is possible to get a good estimation of the most efficient processor distribution by performing a simple sequential run. This is done by simply adding to the input file (in addition to paral_kgb=1) autoparal=1 and max_ncpus=”the maximum number of processors you want”.
max_ncpus is a new variable introduced in Abinit version 7.6.2. Prior to this version, one had to use paral_kgb=-“the maximum number of processors you want” to specify the maximum number of processors.
In order to do that, copy the file tests/tutoparal/Input/tgspw_01.in and the related tgspw_01.files file in your working directory.
# Gold with one vacancy (107 atoms of gold).
rprim  1. 0. 0.
Then run ABINIT on 1 CPU core.
A weight is assigned to each distribution of processors. As indicated in the documentation, you are advised to select a processor distribution with a higher weight. If we just focus on npband and npfft, we see that for 108 processors the recommended distribution is (12x9).
As far as the timing is concerned, the best distributions are then the ones proposed above in section 2.: that is to say the couples (18x6) and (27x4). So the prediction was pretty good.
The last column indicates the convergence of the density (or potential) residual. You can see that this quantity is the smallest when npband is the highest. This result is expected: the convergence is better when the size of the block is as large as possible, so need for re-orthogonalization among band processor pools is minimized. But the best convergence is obtained for the (108x1) distribution… when the worst timing is measured.
So, you face a dilemma. The calculation with the smallest number of iterations (the best convergence) is not the best concerning the timing of one iteration (the best efficiency), and vice versa… So you have to check both of these features for all the processor distributions. On one hand, the timing of one iteration and, on the other hand, the number of iterations needed to converge. The best choice is a compromise between them, not necessarily the independent optima.
In the following we will choose the (27x4) couple, because this one definitively offers more guarantees concerning the convergence and the timing even if the (18x6) one is slightly quicker per electronic step.
Note: you can verify that the convergence is not changed when the npfft value is modified. The same results will be obtained, step by step.
We have seen in the previous section that the best convergence is obtained when the size of the block is the largest. This size is related to the npband input variable. But not only. It is possible to increase the size of the block without increasing drastically the number of band processors. This means that it’s possible to decrease the number of electronic steps without increasing strongly the timing of one electronic step. For systems with peculiar convergence, when some trouble leads the calculation to diverge, this is not just useful but indispensable to converge at all.
Then run ABINIT over 108 cores and copy the output file to tgspw_03.054-02.out. Perform a diff (with vimdiff for example) between the two output files tgspw_03.bandpp1.out and tgspw_03.054-02.out. You can convince yourself that the two calculations (54x2) with bandpp=1 and (27x4) with bandpp=2, give exactly the same convergence. This result is expected, since the sizes of the block are equal (to 54) and the number of FFT processors npfft does not affect the convergence.
It is possible to modify the distribution of processors, without changing the convergence, by reducing npband and increasing bandpp proportionally.
In the previous section, we showed that the convergence doesn’t change if bandpp and npband change in inverse proportions. What about the influence of bandpp if you fix the distribution? Two cases have to be treated separately.
while there are fewer electronic iterations for bandpp=2 (19) than for bandpp=1 (20). If you perform a diff between these two files, you will see that the increase in time is essentially due to the section “zheegv-dsyegv”.
The “zheegv-dsyegv” is a part of the LOBPCG algorithm which is performed in sequential, so the same calculation is done on each processor. In the second calculation, the size of the block being larger (27x2=54) than in the first (27), the computational time of this diagonalization is more expensive. To sum up, the timing of a single electronic step increases by increasing bandpp, but the convergence improves.
Do not increase too much the bandpp value, unless you decrease proportionally npband or if you want to improve the convergence whatever the cost in total timing.
you can see that the timing decreases for bandpp=2 and increases thereafter. This behaviour comes from the FFTs. For bandpp=2, the real wavefunctions are associated in pairs in the complex FFTs, leading to a reduction by a factor of 2 of the timing in this part (you can see this reduction by a diff of the output files). Above bandpp=2, there is longer any gain in the FFTs, whereas some significant losses in computational time appear in the “zheegv-dsyegv” section.
When calculations are performed at the Gamma point, you are strongly encouraged to use bandpp=2… or more if you need to improve the convergence whatever the timing.
is quasi-identical to the one obtained for 1 k-point (69.9 sec, see the output file tgspw_02.027-04.out. This means that a calculation 4 times larger (due to an increase of the number of k-points) gives approximatively the same human time if you parallelize over all the k-points. You have just re-derived a well established result: the scaling (the speedup) is quasi-linear on the k-point level.
When you want to parallelize a calculation, begin by the k-point level, then follow with the band level (up to npband=50 typically) then finish by the FFT level.
We see that the KGB parallelization performs really well, since the wall time spent within vtowfk is approximatively equal: 26409/432 ~ 6372/108. So, the speedup is quasi-linear below vtowfk. The problem comes from parts above vtowfk which are not parallelized and are responsible for the negligible (1-99.xyz)% of time spent in sequential. These parts are no longer negligible when you parallelize over hundreds of processors.
You can also prove this using the percentages rather than the values of the overall times (shown above): the time spent in vtowfk corresponds to 84.8% of the overall time when you don’t parallelize over k-points, and only 70.7% when you parallelize. This means you lose time above vtowfk in this case.
In our case, the part above the loop over k-point in not parallelized by the KGB parallelization. Even if this part is very small, less than 1%, when the biggest part below is strongly (perfectly) parallelized, this small part determines an upper bound for the speedup.
