Introduction to Supercomputing (MCS 572)

• Introduction to Parallel Computing
  • Introduction
    • What is a Supercomputer?
    • Measuring Performance
    • Amdahl’s and Gustafson’s Law
    • Bibliography
    • Exercises
  • Classifications and Scalability
    • Types of Parallel Computers
    • Clusters and Scalability
    • Network Topologies
    • Bibliography
    • Exercises
  • High Level Parallel Processing
    • GPU computing with Maple
    • Multiprocessing in Python
    • Tasking in Ada
    • Performance Monitoring
    • Exercises
• Introduction to Message Passing
  • Basics of MPI
    • One Single Program Executed by all Nodes
    • Initialization, Finalization, and the Universe
    • Broadcasting Data
    • Moving Data from Manager to Workers
    • MPI for Python
    • Bibliography
    • Exercises
  • Using MPI
    • Scatter and Gather
    • Send and Recv
    • Reducing the Communication Cost
    • Point-to-Point Communication with MPI for Python
    • Bibliography
    • Exercises
  • Pleasingly Parallel Computations
    • Ideal Parallel Computations
    • Monte Carlo Simulations
    • SPRNG: scalable pseudorandom number generator
    • Bibliography
    • Exercises
  • Load Balancing
    • the Mandelbrot set
    • Static Work Load Assignment
    • Static work load assignment with MPI
    • Dynamic Work Load Balancing
    • Bibliography
    • Exercises
  • Data Partitioning
    • functional and domain decomposition
    • parallel summation
    • An Application
    • Nonblocking Point-to-Point Communication
    • Exercises
• Introduction to Multithreading
  • Introduction to OpenMP
    • using OpenMP
    • Numerical Integration with OpenMP
    • Bibliography
    • Exercises
  • Introduction to Pthreads
    • the POSIX threads programming interface
    • Using Pthreads
    • The Work Crew Model
    • implementing a critical section with mutex
    • The Dining Philosophers Problem
    • Bibliography
    • Exercises
  • Introduction to the Intel Threading Building Blocks
    • the Intel Threading Building Blocks (TBB)
    • task based programming and work stealing
    • using the parallel_for
    • using the parallel_reduce
    • Bibliography
    • Exercises
• Applications: Sorting, Integration, FFT
  • Parallel Sorting Algorithms
    • Sorting in C and C++
    • Bucket Sort for Distributed Memory
    • Quicksort for Shared Memory
    • Bibliography
    • Exercises
  • Parallel Numerical Integration
    • Numerical Integration
    • Parallel Numerical Integration
    • Bibliography
    • Exercises
  • Parallel FFT and Isoefficiency
    • The Fast Fourier Transform in Parallel
    • Isoefficiency
    • Bibliography
    • Exercises
• Pipelining
  • Pipelined Computations
    • Functional Decomposition
    • Pipeline Implementations
    • Using MPI to implement a pipeline
    • Exercises
  • Pipelined Sorting
    • Pipelines with Intel Threading Building Blocks (TBB)
    • Sorting Numbers
    • Prime Number Generation
    • Exercises
  • Solving Triangular Systems
    • Forward Substitution Formulas
    • Parallel Solving
    • Parallel Solving with OpenMP
    • Exercises
• Synchronized Computations
  • Barriers for Synchronizations
    • Synchronizing Computations
    • The Prefix Sum Algorithm
    • Barriers in Shared Memory Parallel Programming
    • Bibliography
    • Exercises
  • Parallel Iterative Methods for Linear Systems
    • Jacobi Iterations
    • A Parallel Implementation with MPI
    • Gather-to-All with MPI_Allgather
    • Exercises
  • Domain Decomposition Methods
    • Gauss-Seidel Relaxation
    • Parallel Gauss-Seidel with OpenMP
    • Solving the Heat Equation
    • Solving the Heat Equation with PETSc
    • Bibliography
    • Exercises
  • Parallel Gaussian Elimination
    • LU and Cholesky Factorization
    • Blocked LU Factorization
    • The PLASMA Software Library
    • Bibliography
    • Exercises
  • Parallel Numerical Linear Algebra
    • QR Factorization
    • Conjugate Gradient and Krylov Subspace Methods
    • Bibliography
    • Exercises
• Big Data and Cloud Computing
  • Disk Based Parallelism
    • The Disk is the new RAM
    • Roomy: A System for Space Limited Computations
    • Hadoop and the Map/Reduce model
    • Bibliography
    • Exercises
  • Introduction to Hadoop
    • What is Hadoop?
    • Understanding MapReduce
    • Bibliography
    • Exercises
• Architecture and Programming Models for the GPU
  • A Massively Parallel Processor: the GPU
    • Introduction to General Purpose GPUs
    • Graphics Processors as Parallel Computers
    • Bibliography
    • Exercises
  • Evolution of Graphics Pipelines
    • Understanding the Graphics Heritage
    • Programmable Real-Time Graphics
    • Bibliography
  • Programming GPUs with Python: PyOpenCL and PyCUDA
    • PyOpenCL
    • PyCUDA
    • Data Parallelism Model
    • Writing OpenCL Programs
    • Bibliography
    • Programming Guides
• Acceleration with CUDA
  • Introduction to CUDA
    • Our first GPU Program
    • CUDA Program Structure
    • Exercises
  • Data Parallelism and Matrix Multiplication
    • Data Parallelism
    • Code for Matrix-Matrix Multiplication
    • Two Dimensional Arrays of Threads
    • Examining Performance
    • Exercises
  • Device Memories and Matrix Multiplication
    • Device Memories
    • Matrix Multiplication
    • Bibliography
    • Exercises
  • Thread Organization and Matrix Multiplication
    • Thread Organization
    • Matrix Matrix Multiplication
    • Bibliography
    • Exercises
• CUDA Thread Organization
  • Warps and Reduction Algorithms
    • More on Thread Execution
    • Parallel Reduction Algorithms
    • Bibliography
    • Exercises
  • Memory Coalescing Techniques
    • Accessing Global and Shared Memory
    • Memory Coalescing Techniques
    • Avoiding Bank Conflicts
    • Exercises
  • Performance Considerations
    • Dynamic Partitioning of Resources
    • The Compute Visual Profiler
    • Data Prefetching and Instruction Mix
    • Exercises
• Applications and Case Studies
  • Quad Double Arithmetic
    • Floating-Point Arithmetic
    • Quad Double Square Roots
    • Bibliography
    • Exercises
  • Advanced MRI Reconstruction
    • an Application Case Study
    • Acceleration on a GPU
    • Bibliography
  • Concurrent Kernels and Multiple GPUs
    • Page Locked Host Memory
    • Concurrent Kernels
    • Using Multiple GPUs
• Coprocessor Acceleration
  • The Intel Xeon Phi Coprocessor
    • Many Integrated Core Architecture
    • Programming the Intel Xeon Phi Coprocessor with OpenMP
    • Bibliography

Indices and tables

• Index
• Module Index
• Search Page