The first program "cost_scatter.c" shows how bad things can get.
the cost of scattering a 2520-by-2520 matrix of random doubles over p processors
p size/p total wt(p1) wt(p2) wt(p3) wt(p4) wt(p5) wt(p6) wt(p7) wt(p8) wt(p9)
2 1260 2.619 2.685
3 840 3.328 1.920 3.371
4 630 3.663 1.522 2.604 3.695
5 504 3.873 1.303 2.169 3.033 3.898
6 420 4.020 1.150 1.873 2.596 3.316 4.047
7 360 4.129 1.048 1.667 2.288 2.909 3.533 4.152
8 315 4.201 0.962 1.502 2.043 2.584 3.130 3.678 4.220
9 280 4.264 0.898 1.379 1.867 2.348 2.832 3.315 3.795 4.281
10 252 3.910 0.843 1.275 1.710 2.143 2.579 3.012 3.445 3.880 3.916
Scattering a 2520-by-2520 matrix over 10 processors means that every processor gets 252 rows.
Observe:
1) The total scatter time is about the time it takes to reach the last processor.
2) The wall time wt(p1) of processor 1 decreases as the size of the submatrices decreases.
We computed the startup or latency time taking the data from the column
wt(p1).
A variant of this program would be to use MPI_Send/MPI_Recv instead of MPI_Scatter. It is left as an exercise.
The second program of the day is "cost_fan.c" which cuts the time in half for eight processors:
prompt] mpirun -np 8 cost_fan Processor 0 is sending to 1. Processor 0 is sending to 2. Processor 0 is sending to 4. Total wall time = 2.030882 seconds on 8 processors Wall time on processor 1 = 2.371648 seconds. Wall time on processor 2 = 2.657336 seconds. Wall time on processor 3 = 2.423442 seconds. Wall time on processor 4 = 2.047735 seconds. Wall time on processor 5 = 2.393837 seconds. Wall time on processor 6 = 2.679017 seconds. Wall time on processor 7 = 2.447044 seconds. Processor 2 is receiving from 0. Processor 2 is sending to 6. Processor 4 is receiving from 0. Processor 3 is receiving from 1. Processor 3 is sending to 7. Processor 7 is receiving from 3. Processor 1 is receiving from 0. Processor 1 is sending to 3. Processor 1 is sending to 5. Processor 6 is receiving from 2. Processor 5 is receiving from 1. prompt]Notice that the second program cheats a bit: we are only broadcasting the top rows of the random matrix and it does not distribute the entire matrix as the first program.
We ended the lecture by observing that a processor running at a clock speed of 2.4Ghz is able to perform 2.4 10^9 flops (floating operations per second). With n = 2520, or 10^3, an algorithm which requires O(n^3) operations would finish in a second.
Bibliography