Unit 6: Introduction to GPU Architecture

I. Orientation: The GPU Design Principle

A Graphics Processing Unit trades the CPU's few powerful cores for thousands of simple cores, optimising throughput over latency (the model matured with NVIDIA's CUDA launch in 2007). Every later section rests on this throughput-oriented, data-parallel philosophy.

• SIMT execution: Single Instruction, Multiple Threads — one instruction is broadcast to a warp of 32 threads that execute in lockstep on distinct data elements.
• Memory hierarchy: registers → shared memory (per block) → L2 cache → global HBM/GDDR; bandwidth (up to ~3 TB/s on HBM3) matters more than absolute latency.
• Occupancy over speed: thousands of resident threads hide memory latency by context-switching warps in a single cycle rather than stalling.
• Amdahl's constraint: speedup is bounded by the serial fraction; a workload 95% parallel caps at 20× regardless of core count.
• Heterogeneous role: the GPU is a co-processor; the host CPU dispatches kernels and manages control flow.

II. Nvidia Case Study

A. The GPU as a general-purpose accelerator

NVIDIA converted the fixed-function graphics pipeline into a programmable parallel machine, defining the modern GPGPU market.

• CUDA model: kernels launched as a grid of blocks of threads; <<<blocks, threads>>> syntax maps software hierarchy onto hardware Streaming Multiprocessors (SMs).
• Architectural generations: Tesla (2007) → Fermi → Kepler → Pascal → Volta (2017, first Tensor Cores) → Ampere (A100) → Hopper (H100) → Blackwell (2024).
• Tensor Cores: dedicated matrix-multiply-accumulate units performing D = A×B + C on 4×4 tiles per clock, accelerating deep-learning GEMM by an order of magnitude.
• Streaming Multiprocessor (SM): the core building block — holds CUDA cores, warp schedulers, register file and shared memory; an H100 has 132 SMs.
• Interconnect: NVLink and NVSwitch give ~900 GB/s GPU-to-GPU bandwidth, letting many GPUs act as one for large-model training.

B. Significance

• AI dominance: the H100/Blackwell line underpins most large-language-model training clusters.
• Ecosystem lock-in: cuDNN, cuBLAS and TensorRT libraries make CUDA the de facto parallel-computing standard.

III. Introduction to Supercomputer

A. Definition and purpose

A supercomputer aggregates thousands of nodes to reach the highest sustained computational rate, measured in FLOPS and ranked by the LINPACK benchmark on the TOP500 list.

• Performance units: petaFLOPS (10¹⁵) and exaFLOPS (10¹⁸) floating-point operations per second.
• Exascale milestone: Frontier (Oak Ridge, 2022) first exceeded 1 exaFLOP, using AMD EPYC CPUs plus MI250X GPUs.
• Architecture: massively parallel clusters of CPU+GPU nodes joined by low-latency fabric (InfiniBand / Slingshot).
• Programming: MPI for inter-node message passing, OpenMP/CUDA for intra-node parallelism.

B. Design constraints

• Power wall: Frontier draws ~21 MW; efficiency is tracked in GFLOPS/watt on the Green500 list.
• Cooling and reliability: warm-water liquid cooling; checkpoint/restart handles frequent node failures at scale.
• Workloads: climate modelling, molecular dynamics, nuclear simulation, and increasingly AI training.

IV. Introduction to Qubits and Quantum Computing

A. The quantum bit

A qubit stores information in a quantum superposition, enabling algorithmic parallelism impossible for classical bits.

• State vector: written |ψ⟩ = α|0⟩ + β|1⟩ where α, β are complex amplitudes.

|α|² + |β|² = 1      (normalisation)
P(0) = |α|²,  P(1) = |β|²   (measurement probabilities)

• Superposition: n qubits represent 2ⁿ states simultaneously — 300 qubits exceed the atom count of the universe.
• Entanglement: correlated qubits (e.g. Bell state (|00⟩+|11⟩)/√2) share state non-locally.
• Measurement: collapses the superposition to one classical outcome, destroying the amplitudes.

B. Computation and hardware

• Quantum gates: unitary operations — Hadamard (H) creates superposition, CNOT entangles two qubits.
• Algorithms: Shor's factoring (threatens RSA), Grover's search (√N speedup over N).
• Physical realisations:
  1. Superconducting: IBM, Google — fast gates, needs ~15 mK dilution refrigeration.
  2. Trapped-ion: IonQ — longer coherence, slower gates.
• Limitation — decoherence: fragile states demand quantum error correction, spending many physical qubits per logical qubit.

V. Latest Technology and Trends in Computer Architecture

A. Beyond the classic scaling laws

With Dennard scaling ended and Moore's Law slowing, architects pursue specialisation and packaging innovation.

• Chiplets: a large die is split into smaller chiplets joined on an interposer, raising yield — AMD's EPYC uses this "Zen" approach.
• Domain-specific accelerators: Google TPU, NPUs and Tensor Cores replace general cores for AI matrix workloads.
• 3D stacking: HBM stacks DRAM vertically via through-silicon vias; AMD's 3D V-Cache stacks cache atop logic.
• Heterogeneous integration: big.LITTLE-style mixing of performance and efficiency cores on one die.
• Near-memory / in-memory computing: moving compute toward data to beat the memory wall.

B. Systemic drivers

• Energy efficiency: performance-per-watt now dominates raw clock speed.
• RISC-V rise: open, royalty-free ISA enabling custom architectures.
• CXL interconnect: Compute Express Link enables coherent memory pooling across CPU, GPU and accelerators.

VI. Next Generation Processors Architecture

A. Emerging processor paradigms

Future processors combine heterogeneous compute blocks on advanced packaging rather than shrinking a single monolithic core.

• APU / SoC fusion: CPU, GPU and NPU on one package (Apple M-series, AMD Instinct MI300 combining CPU+GPU chiplets).
• Dataflow and wafer-scale: Cerebras WSE integrates an entire wafer (~900,000 cores) to eliminate off-chip communication.
• Photonic interconnect: optical links to cut communication energy between chiplets.
• Neuromorphic: Intel Loihi mimics spiking neurons for event-driven, ultra-low-power inference.

B. AI-centric direction

• Dedicated NPUs: on-device inference engines rated in TOPS (tera-operations/second).
• Reduced precision: FP8, INT4 and bfloat16 formats trade numeric range for throughput and lower memory footprint.

VII. Microarchitecture

A. From ISA to hardware realisation

Microarchitecture is the concrete organisation implementing an instruction-set architecture — the same ISA (x86, ARM) can have many microarchitectures.

• Pipelining: overlap fetch, decode, execute, memory, write-back so one instruction completes per cycle ideally.
• Superscalar issue: multiple execution ports dispatch several instructions per cycle.
• Out-of-order execution: a reorder buffer and reservation stations run ready instructions early, then retire in program order.
• Branch prediction: speculative execution guided by predictors; misprediction flushes the pipeline (the root of Spectre-class attacks).
• Register renaming: maps architectural to physical registers, removing false (WAR/WAW) dependencies.

B. Memory and hazards

• Cache hierarchy: L1 (per-core, ~ns), L2, shared L3; exploits temporal and spatial locality.
• Hazards:
  1. Data hazard: an instruction needs a result not yet produced — solved by forwarding.
  2. Control hazard: branch outcome unknown — solved by prediction and speculation.
• SIMD units: vector extensions (AVX-512, ARM SVE) process multiple data lanes per instruction.

VIII. Latest Processor for Smartphone or Tablet and Desktop

A. Mobile and desktop compared

Modern high-end chips share heterogeneous, big.LITTLE-style designs but diverge sharply in power envelope and I/O scope.

1. Smartphone / tablet SoCs:

   • Apple A-series & M-series: ARM-based, TSMC 3 nm; unified memory shared by CPU/GPU/NPU; on-die Neural Engine (~35 TOPS).
   • Qualcomm Snapdragon 8 Gen 3 / Elite: Adreno GPU plus Hexagon NPU; integrated 5G modem.
   • Power envelope: ~5–15 W, fanless, prioritising battery life and idle efficiency.
   • Integration: modem, ISP, DSP and NPU on one SoC.

2. Desktop CPUs:

   • Intel Core (Meteor/Arrow Lake): disaggregated tiles (compute, GPU, SoC, I/O) mixing P-cores and E-cores via the Thread Director scheduler.
   • AMD Ryzen (Zen 4/5): chiplet CCDs plus separate I/O die; optional 3D V-Cache variants for gaming.
   • Power envelope: 65–250 W with active cooling, favouring peak multi-threaded throughput.
   • Expandability: PCIe lanes, discrete GPU support, socketed DIMMs.

B. Convergence trend

• Shared ISA path: ARM increasingly rivals x86 on the desktop (Apple M-series, Snapdragon X Elite laptops).
• NPU standardisation: dedicated AI blocks now appear across both classes, measured in TOPS, reflecting the unit-wide shift toward specialised, heterogeneous parallel silicon.