Introduction to Supercomputing (MCS 572)

• Preface
  • Three Different Types of Parallelism
  • Programming Languages
  • Bibliography
• Introduction to Parallel Computing
  • Introduction
    • What is a Supercomputer?
    • Measuring Performance
    • Amdahl’s and Gustafson’s Law
    • Bibliography
    • Exercises
  • Classifications and Scalability
    • Types of Parallel Computers
    • Clusters and Scalability
    • Network Topologies
    • Bibliography
    • Exercises
  • High Level Parallel Processing
    • High-Level Parallel Programming
    • Multiprocessing in Python
    • Multithreading with Julia
    • Tasking in Ada
    • Performance Monitoring
    • Bibliography
    • Exercises
• Introduction to Message Passing
  • Basics of MPI
    • One Single Program Executed by all Nodes
    • Initialization, Finalization, and the Universe
    • Broadcasting Data
    • Moving Data from Manager to Workers
    • MPI for Python
    • MPI wrappers for Julia
    • Bibliography
    • Exercises
  • Using MPI
    • Scatter and Gather
    • Send and Recv
    • Reducing the Communication Cost
    • Point-to-Point Communication with MPI for Python
    • Point-to-Point Communication with the MPI wrappers in Julia
    • Bibliography
    • Exercises
  • Pleasingly Parallel Computations
    • Ideal Parallel Computations
    • Monte Carlo Simulations
    • SPRNG: scalable pseudorandom number generator
    • Bibliography
    • Exercises
  • Load Balancing
    • the Mandelbrot set
    • Granularity
    • Static Work Load Assignment
    • Static work load assignment with MPI
    • an implementation with mpi4py
    • Dynamic Work Load Balancing
    • probing in Python and Julia
    • Scalability
    • Bibliography
    • Exercises
  • Handson Supercomputing
    • working on a fast workstation
    • using a real supercomputer
  • Data Partitioning
    • functional and domain decomposition
    • parallel summation
    • An Application
    • Nonblocking Point-to-Point Communication
    • Exercises
• Introduction to Threading and Tasking
  • Introduction to OpenMP
    • programming shared memory parallel computers
    • multithreading in Julia
    • the OpenMP Application Program Interface
    • using OpenMP
    • Numerical Integration with OpenMP
    • Bibliography
    • Exercises
  • The Crew of Threads Model
    • Multithreaded Processes
    • The Work Crew Model
    • A Crew of Workers with Julia
    • Processing a Job Queue
    • Processing the Jobs with OpenMP
    • The POSIX Threads Programming Interface
    • Implementing a Critical Section with mutex
    • The Dining Philosophers Problem
    • Bibliography
    • Exercises
  • Tasking with OpenMP
    • Parallel Recursive Functions
    • Parallel Recursive Quadrature
    • Bernstein’s Conditions
    • Task Dependencies
    • Parallel Blocked Matrix Multiplication
    • Bibliography
    • Exercises
  • Tasking with Julia
    • Parallel Recursive Functions
    • Parallel Recursive Quadrature
    • Parallel Merge Sort
    • Basic Linear Algebra Subprograms
    • Exercises
  • Evaluating Parallel Performance
    • Metrics
    • Isoefficiency
    • Task Graph Scheduling
    • The Roofline Model
    • Bibliography
  • Work Stealing
    • Work Stealing Simulated by a Julia Program
    • Multithreading in Python with Numba
    • Multithreading in Python with Parsl
    • the Intel Threading Building Blocks (TBB)
    • using the parallel_for
    • using the parallel_reduce
    • Bibliography
    • Exercises
• Acceleration with Graphics Processing Units
  • A Massively Parallel Processor: the GPU
    • Introduction to General Purpose GPUs
    • Graphics Processors as Parallel Computers
    • Bibliography
    • Exercises
  • Programming GPUs with PyCUDA and Julia
    • Data Parallelism
    • Matrix Matrix Multiplication
    • PyCUDA
    • Vendor Agnostic GPU Computing in Julia
    • Bibliography
    • Exercises
  • Introduction to CUDA
    • Our first GPU Program
    • CUDA Program Structure
    • using CUDA.jl
    • Exercises
  • Data Parallelism and Matrix Multiplication
    • Data Parallelism
    • Code for Matrix-Matrix Multiplication
    • Two Dimensional Arrays of Threads
    • Examining Performance
    • using CUDA.jl and Metal.jl
    • Exercises
  • Device Memories and Matrix Multiplication
    • Device Memories
    • Matrix Multiplication
    • using shared memory with CUDA.jl
    • Bibliography
    • Exercises
  • Thread Organization and Matrix Multiplication
    • Thread Organization
    • Matrix Matrix Multiplication
    • Submatrices with Threads in CUDA.jl
    • Thread Synchronization
    • Bibliography
    • Exercises
  • Warps and Reduction Algorithms
    • More on Thread Execution
    • Parallel Reduction Algorithms
    • Julia Defined Kernels
    • Bibliography
    • Exercises
• Review for the Midterm Exam
  • Four Sample Questions
    • Scaled Speedup
    • Network Topologies
    • Task Graph Scheduling
    • Compute Bound or Memory Bound
  • Fall 2024 Midterm Questions
    • Question 1 : Isoefficiency
    • Question 2 : the roofline model
    • Question 3 : tasking for enumeration
    • Question 4 : CGMA ratio
• Pipelining and Synchronized Computations
  • Pipelined Computations
    • Functional Decomposition
    • Loop Unrolling
    • Pipeline Implementations
    • Using MPI to implement a pipeline
    • Exercises
  • Pipelined Sorting, Sieving, Substitution
    • Sorting Numbers
    • Prime Number Generation
    • Solving Triangular Systems
    • Bibliography
    • Exercises
  • Solving Triangular Systems
    • Ill Conditioned Matrices and Quad Doubles
    • On a Parallel Shared Memory Computer with OpenMP
    • Accelerated Back Substitution
    • Bibliography
    • Exercises
  • Barriers for Synchronizations
    • Synchronizing Computations
    • The Prefix Sum Algorithm
    • Brent’s Theorem
    • Bibliography
    • Exercises
  • Parallel Iterative Methods for Linear Systems
    • Jacobi Iterations
    • A Parallel Implementation with MPI
    • Gather-to-All with MPI_Allgather
    • Strip Partitioning and Reduce Barriers in Julia
    • Exercises
  • Domain Decomposition Methods
    • Gauss-Seidel Relaxation
    • Parallel Gauss-Seidel with OpenMP
    • Solving the Heat Equation
    • Solving the Heat Equation with PETSc
    • Bibliography
    • Exercises
  • Memory Coalescing Techniques
    • Accessing Global and Shared Memory
    • Memory Coalescing Techniques
    • Avoiding Bank Conflicts
    • Exercises
  • Introduction to Tensor Cores
    • High Throughput Computing
    • Volta, Ampere, Hopper Architectures
    • Simple Matrix Multiplication
    • Bibliography
    • Exercises
  • Performance Considerations
    • Dynamic Partitioning of Resources
    • The Compute Visual Profiler
    • Data Prefetching and Instruction Mix
    • Thread Coarsening
    • Exercises
• Applications
  • Parallel FFT and Sorting
    • The Fast Fourier Transform in Parallel
    • Bucket Sort for Distributed Memory
    • Quicksort for Shared Memory
    • Radix Sort for GPU acceleration
    • Bibliography
    • Exercises
  • Parallel Gaussian Elimination
    • LU and Cholesky Factorization
    • Blocked LU Factorization
    • The PLASMA Software Library
    • The cuSOLVER Library
    • Bibliography
    • Exercises
  • GPU Accelerated QR
    • Householder Reflectors
    • Multiple Doubles on Graphics Processing Units
    • Accelerated Blocked Householder QR
    • Bibliography
    • Exercises
  • Advanced MRI Reconstruction
    • an Application Case Study
    • Acceleration on a GPU
    • Bibliography
  • GPU Accelerated Newton’s Method for Taylor Series
    • Problems Statement and Overview
    • Data Staging for Evaluation and Differentiation
    • Accelerated Newton’s Method
    • Bibliography
• Review for the Final Exam
  • Four Sample Questions
    • Limits on Speedup
    • Space Time Diagram
    • Registers and Shared Memory
    • Convolutions
  • Fall 2024 Final Exam Questions
    • Question 1 : Isoefficiency
    • Question 2 : Multitasked Radix Sort
    • Question 3 : Divide-and-Conquer Polynomial Evaluation
    • Question 4 : Pipelined Function Evaluation
    • Question 5 : Data Staging
    • Question 6 : Resource Limitations

Indices and tables

• Index

• Module Index

• Search Page