scalability -- message passing computing

As promised we started the meeting by deriving Gustafson's Law: if s is the fraction of operations which are executed sequentially when executing using p processors (i.e.: the fraction of the time only one processor is working, the rest is idle), then the scaled speedup is bounded by p + (1-p)*s. The scaled speedup is the ratio of the time a sequential program would take over the execution time of a parallel program. Note that the formulation of Gustafson's law in the textbook has a type error: the formula is timeless.

We defined the computation/communication ratio. The two basic types of parallel computers are shared memory and distributed memory. In the second half of the course we will learn how to use OpenMP to program shared memory computers. Because p^2 switches are needed in the crossbar switch network to link processors with memory modules, only a relatively modest number of processors are available in a shared memory computer. A distributed memory computer can accommodate a larger number of processors, although the communication overhead is larger. Next week we will start using MPI (Message Passing Interface) to program a distributed memory multicomputer.

We defined bandwidth, latency (startup time or message latency, network latency, and communication latency), the diameter of a network, bisection width and bisection bandwidth. We computed the diameter of a mesh network and a hypercube.

Bibliography