MCS 572: Introduction to Supercomputing
The goal of the course is to study parallel algorithms and
their implementation on supercomputers.
The evaluation of the course consists mainly of homework and
computer projects. Exams are scheduled as preparation for the
computational science prelim.
The recommended (not required) textbooks are
- "Parallel Programming. Techniques
and Applications Using Networked Workstations and Parallel
Computers"
by Barry Wilkinson and Michael Allen, Pearson Prentice Hall,
second edition, 2005.
- "Programming Massively Parallel Processors.
A Hands-on Approach"
by David B. Kirk and Wen-mei W. Hwu,
Elsevier/Morgan Kaufmann Publishers, 2010; second edition, 2013.
Slides for the lectures will be posted below.
0. Introduction
1. Distributed Memory Parallel Computing
2. Shared Memory Parallel Computing
- L-10 02/01/23: Introduction to OpenMP.
To write parallel programs for shared memory computers
we can use OpenMP.
Slides.
Simple C programs:
hello_openmp0.c,
hello_openmp1.c,
hello_openmp2.c,
comptrap.c, and
comptrap_omp.c.
Prerecorded lecture:
youtube link.
- L-11 02/03/23: The Work Crew Model.
We define dynamic work load assignments for parallel
shared memory computers, using OpenMP, followed by
a brief introduction to pthreads and Julia.
Slides.
Simple C programs:
jobqueue.h,
jobqueue.c,
test_jobqueue.c,
run_jobqueue_omp.c,
hello_pthreads.c, and
process_jobqueue.c.
Some Julia programs:
hellothreads.jl and
workcrew.jl
Prerecorded lecture:
youtube link.
- L-12 02/06/23: Tasking with OpenMP.
Recursive functions can run in parallel.
Bernstein's conditions are applied for a dependency analysis.
We end with a parallel blocked matrix computation.
Slides.
Simple C programs:
fib.c,
fibomp.c,
comptraprec.c,
comptraprecomp.c, and
matmulomp.c.
Prerecorded lecture:
youtube link.
- L-13 02/08/23: Tasking with Julia.
We redo the last lecture, but now with Julia.
Slides.
Simple Julia programs:
showargs.jl,
showthreads.jl,
fibmt.jl,
traprulerec.jl,
traprulerecmt.jl,
mergesortmt.jl, and
matmatmulmt.jl.
Prerecorded lecture:
youtube link.
- L-14 02/10/23: Evaluating Parallel Performance.
We look at metrics, isoefficiency, critical path analysis,
and the roofline model
to evaluate the performance of parallel algorithms.
Slides.
Prerecorded lecture:
youtube link.
- L-15 02/13/23: Work Stealing.
For job assignment, we can view work stealing as a hybrid between
static and dynamic work assignment, as defined by a Julia program.
The Threading Building Blocks provide a high level programming model
for C++ programmers.
Slides.
A simple Julia program:
worksteal.jl
A Python script with Numba:
montecarlo4pi.py
C++ programs:
hello_task_group.cpp,
powers_serial.cpp,
powers_tbb.cpp, and
parsum_tbb.cpp
Prerecorded lecture:
youtube link.
3. Acceleration with Graphics Processing Units
- L-16 02/15/23: a Massively Parallel Processor: the GPU.
We introduce the Graphics Processing Unit for massive parallel computing,
applying data parallelism.
Slides.
Prerecorded lecture:
youtube link.
- L-17 02/17/23: the Evolution of Graphics Pipelines.
The historical evolution of graphics processors explains the current
execution model of GPUs.
Slides.
Prerecorded lecture:
youtube link.
- L-18 02/20/23: programming GPUs with PyCUDA and CUDA.jl.
We can launch kernels in Python with PyCUDA and in Julia with CUDA.jl.
Slides.
a simple Python script:
matmatmulcuda.py
a simple Julia program:
gpuadd.jl
Prerecorded lecture:
youtube link.
- L-19 02/22/23: Introduction to CUDA. Writing CUDA programs is
a 5-step process.
Slides.
Our first CUDA programs:
cudaDoubleComplex.cu and
runCudaComplexSqrt.cu.
a simple Julia program:
gpuadd3.jl
Prerecorded lecture:
youtube link.
- L-20 02/24/23: Data Parallelism and Matrix Multiplication.
The CUDA program structure is illustrated on the matrix-matrix
multiplication problem.
Slides.
A simple C program:
matmatmul0.c;
and two simple CUDA programs:
matmatmul1.cu and
matmatmul2.cu
and an even simpler Julia program:
mmmulcuda2.jl.
Prerecorded lecture:
youtube link.
- L-21 02/27/23: Device Memories and Matrix Multiplication.
Understanding different memories matters in the calculation
of the expected performance of kernels.
We define the Compute to Global Memory Access ratio.
Slides.
Prerecorded lecture:
youtube link.
- L-22 03/01/23: Thread Organization and Matrix Multiplication.
We look at the matrix multiplication problem from the perspective
of the organization of threads.
Slides.
A simple CUDA program:
organization.cu.
some Julia programs with Metal.jl and CUDA.jl:
gpuadd1metal.jl,
mmmulmetal.jl, and
organization.jl.
Prerecorded lecture:
youtube link.
- L-23 03/03/23: review for the midterm exam.
Slides.
Prerecorded lecture:
youtube link.
- L-24 03/05/23:
questions on the midterm exam.
4. Pipelining and Synchronized Computations
- L-25 03/08/23: Pipelined Computations.
As in the manufactoring of cars, arranging the stages in a computation
along a pipeline speeds up the computations.
Slides.
A Julia program:
leibnizunrolled.jl.
Some simple C programs:
pipe_ring.c and
pipe_sum.c.
Prerecorded lecture:
youtube link.
- L-26 03/10/23: Pipelined Sorting and Sieving.
We illustrate pipelined sorting algorithms with MPI.
Type 2 pipelines are introduced with sieving for primes.
Type 3 pipelines are illustrated by triangular linear systems.
Slides.
One simple C program:
pipe_sort.c
Prerecorded lecture:
youtube link.
- L-27 03/13/23: Solving Triangular Systems.
We consider the solving of triangular linear systems,
using first a high level OpenMP implementation and
then a GPU accelerated solver with CUDA, both in multiple double precision.
Slides.
Some simple C++ programs:
qd4sqrt2.cpp,
triangularsolve.cpp,
triangularsolve_qd.cpp (with QD), and
triangularsolve_qd_omp.cpp
(with QD and OpenMP).
Prerecorded lecture:
youtube link.
- L-28 03/15/23: Barriers for Synchronizations.
For message passing, we distinguish between a linear, a tree, and
a butterfly barrier. We end with the PRAM model and Brent's theorem.
Slides.
Two simple C programs with MPI:
use_sendrecv.c and
prefix_sum.c.
Prerecorded lecture:
youtube link.
- L-29 03/17/23: Parallel Iterative Methods for Linear Systems.
We look at Jacobi's method. Slides.
Some C programs:
use_allgather.c,
jacobi.c, and
jacobi_mpi.c.
Some Julia programs:
jacobi.jl,
mtmatvec.jl,
mtreduce.jl, and
mtjacobi.jl.
Prerecorded lecture:
youtube link.
- L-30 03/27/23: Domain Decomposition Methods.
We continue our investigations into parallel synchronized iterations,
discuss a parallel implementation of the Gauss-Seidel method
and a simple time stepping method for the heat equation.
Slides.
Two simple C programs:
gauss_seidel.c, and
gauss_seidel_omp.c.
Prerecorded lecture:
youtube link.
- L-31 03/29/23: Warps and reduction algorithms.
We discuss the development of a kernel with less thread divergence
to reduce a sequence of numbers.
Slides.
Two simple Julia programs:
gpusum32metal.jl, and
gpusum32cuda.jl.
Prerecorded lecture:
youtube link.
- L-32 03/31/23: Memory Coalescing Techniques.
Data staging algorithms arrange data so adjacent threads access
adjacent memory locations.
Slides.
One simple Julia program:
gpupwr32cuda.jl.
Prerecorded lecture:
youtube link.
- L-33 04/03/23: Performance Considerations and Tensor Cores.
We define the notion of a performance cliff and end with
a short introduction to NVIDIA Tensor Cores.
Slides.
Prerecorded lecture:
youtube link.
6. Applications
- L-34 04/05/23: Parallel FFT and Sorting.
FFT and sorting both have quasi linear complexity.
Slides.
Using FFTW in C:
fftw_use.c,
fftw_timing.c, and
fftw_timing_omp.c.
Simple C programs using qsort:
use_qsort.c,
time_qsort.c,
part_qsort.c, and
part_qsort_omp.c.
simple C++ programs using STL sort:
use_stl_sort.cpp,
time_stl_sort.cpp, and
tbb_sort.cpp.
Prerecorded lecture:
youtube link.
- L-35 04/07/23: parallel Gaussian elimination.
We consider tiled Cholesky and LU factorizations.
Slides.
Prerecorded lecture:
youtube link.
- L-36 04/10/23: GPU accelerated QR.
The blocked Householder QR is rich in matrix matrix multiplications
and well suited for GPU acceleration.
GPUs capable of teraflop performance can compensate for the cost overhead
of quad double arithmetic.
Slides.
a Julia program
houseQR.jl,
and corresponding Jupyter notebook:
houseQR.ipynb,
Prerecorded lecture:
youtube link.
- L-37 04/12/23: Case Study: Advanced MRI Reconstruction.
Slides.
Prerecorded lecture:
youtube link.
- L-38 04/14/23: Big Data Computing: disk parallelism and cloud computing.
Slides.
Prerecorded lecture:
youtube link.
- L-39 04/17/23: Concurrent Kernels and Multiple GPUs.
Slides.
some CUDA programs:
pinnedmemorycheck.cu,
pinnedmemoryuse.cu,
concurrent.cu, and
count_devices.cu
Prerecorded lecture:
youtube link.
- L-40 04/19/23: GPU accelerated Newton's method for Taylor series.
Slides.
Prerecorded lecture:
youtube link.
L-41 04/21/23: a second review. Slides.
Prerecorded lecture:
youtube link.