Unit 6: Introduction to GPU Architecture - Subjective Questions

A CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) differ significantly in their design philosophy:

CPU Architecture:

• Optimized for low-latency execution of sequential tasks.
• Contains a few powerful cores (typically 4–16) with large caches.
• Devotes significant die area to control logic and branch prediction.
• Excellent at handling complex, unpredictable control flow.

GPU Architecture:

• Optimized for high throughput rather than low latency.
• Contains thousands of smaller, simpler cores (e.g., CUDA cores).
• Devotes most die area to ALUs (arithmetic logic units) rather than control and cache.
• Uses a SIMT (Single Instruction Multiple Threads) execution model.

Why GPUs suit data-parallel workloads:

• Data-parallel problems apply the same operation across large data sets (e.g., matrix multiplication, image processing).
• GPUs hide memory latency by switching between thousands of threads rather than relying on large caches.
• The massive number of cores allows simultaneous processing of many data elements.

In summary, CPUs are latency-oriented while GPUs are throughput-oriented, making GPUs ideal for parallel numeric computation.

NVIDIA GPUs are built on a hierarchical, massively parallel architecture:

1. Streaming Multiprocessors (SMs):

• The fundamental building blocks of an NVIDIA GPU.
• Each SM contains multiple CUDA cores, warp schedulers, register files, and shared memory.
• A GPU consists of many SMs working in parallel.

2. CUDA Cores:

• The basic execution units that perform integer and floating-point operations.
• A high-end GPU can contain thousands of CUDA cores.

3. Warp Execution (SIMT):

• Threads are grouped into warps (typically 32 threads).
• All threads in a warp execute the same instruction simultaneously in SIMT fashion.

4. Memory Hierarchy:

• Registers: Fastest, private per thread.
• Shared Memory / L1 Cache: Shared among threads in a block, low latency.
• L2 Cache: Shared across SMs.
• Global Memory (DRAM): Largest but highest latency.

5. Specialized Units (modern NVIDIA GPUs):

• Tensor Cores for accelerating AI/deep learning matrix operations.
• RT Cores for real-time ray tracing.

This hierarchy allows NVIDIA GPUs to achieve very high computational throughput for graphics and general-purpose computing (GPGPU) via CUDA.

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA that allows developers to use GPUs for general-purpose computing (GPGPU).

CUDA Programming Model Hierarchy:

• Thread: The smallest unit of execution. Each thread runs the same kernel function on different data.
• Block (Thread Block): A group of threads that can cooperate by sharing data via shared memory and can synchronize using barriers. Threads within a block are identified by threadIdx.
• Grid: A collection of thread blocks that execute the same kernel. Blocks are identified by blockIdx.

Execution Flow:

1. Host (CPU) allocates memory and copies data to the device (GPU).
2. Host launches a kernel with a specified grid and block configuration:

kernelFunction<<<numBlocks, threadsPerBlock>>>(args);

3. GPU executes the kernel in parallel across all threads.
4. Results are copied back from device to host.

Key Benefits:

• Scalability across GPUs of different sizes.
• Efficient use of the SIMT model.
• Explicit control over memory hierarchy for optimization.

CUDA enables acceleration of scientific computing, AI, and simulation workloads.

A supercomputer is a computer with an extremely high level of computational performance compared to a general-purpose computer, capable of performing quadrillions of calculations per second (measured in FLOPS – Floating Point Operations Per Second).

Key Characteristics:

• Massive Parallelism: Uses thousands to millions of processing cores working together.
• High-speed Interconnects: Nodes are linked with low-latency, high-bandwidth networks (e.g., InfiniBand).
• Enormous Memory and Storage: Petabytes of storage and large distributed memory.
• Specialized Cooling: Often liquid-cooled due to high power consumption.
• Measured in PetaFLOPS/ExaFLOPS: Modern supercomputers reach the exascale ( FLOPS).

Typical Applications:

• Weather forecasting and climate modeling.
• Computational fluid dynamics and aerospace simulations.
• Molecular modeling and drug discovery.
• Nuclear simulations and physics research.
• AI/Machine learning training at scale.
• Genome sequencing and bioinformatics.

Supercomputers are ranked worldwide by the TOP500 list based on the LINPACK benchmark.

Modern supercomputers use a massively parallel, cluster-based architecture built from many interconnected computing nodes.

Architectural Components:

1. Compute Nodes:

• Each node contains multiple CPUs and often GPUs or other accelerators.
• Nodes have local memory (distributed memory model).

2. Clusters:

• A cluster is a collection of nodes working together as a single system.
• Workloads are divided among nodes and processed in parallel.
• Enables scalability — adding more nodes increases performance.

3. Interconnects:

• High-speed networks (e.g., InfiniBand, custom fabrics) link nodes together.
• Provide low latency and high bandwidth for message passing (via MPI).
• The efficiency of interconnects is critical because inter-node communication can become a bottleneck.

4. Parallel File System:

• Shared, high-throughput storage (e.g., Lustre, GPFS) accessible by all nodes.

5. Software Stack:

• Job schedulers, MPI libraries, and parallel programming frameworks.

Role in Performance:

• Clusters provide raw computational scale.
• Interconnects ensure nodes can exchange data quickly, keeping cores busy.
• Together they enable coordinated solving of large problems that no single machine could handle.

Balanced compute, memory, and communication are essential to avoid performance bottlenecks.

A qubit (quantum bit) is the fundamental unit of information in quantum computing, analogous to the classical bit but governed by the laws of quantum mechanics.

Classical Bit vs Qubit:

• A classical bit can be in one of two states: 0 or 1.
• A qubit can be in state , , or any superposition of both.

Superposition:

• A qubit can exist in a linear combination of both states simultaneously:
  
  where and are complex probability amplitudes with .
• Upon measurement, the qubit collapses to with probability or with probability .

Entanglement:

• A quantum phenomenon where two or more qubits become correlated such that the state of one instantly influences the state of the other, regardless of distance.
• Measuring one entangled qubit immediately determines the outcome of the other.

Significance:

• qubits can represent states simultaneously, enabling massive parallelism.
• Superposition and entanglement are the foundation of quantum speedups in algorithms like Shor's and Grover's.

Quantum computing is a computing paradigm that uses principles of quantum mechanics — such as superposition, entanglement, and interference — to process information using qubits instead of classical bits.

Quantum vs Classical Computing:

Key Principles Used:

• Superposition: Represent multiple states at once.
• Entanglement: Correlate qubits for coordinated computation.
• Quantum Interference: Amplify correct answers, cancel wrong ones.

Potential Applications:

• Cryptography (Shor's algorithm for factoring).
• Optimization problems.
• Drug discovery and molecular simulation.
• Machine learning acceleration.
• Search (Grover's algorithm).

Challenges:

• Decoherence: Qubits lose their quantum state quickly.
• Error rates: Require complex quantum error correction.
• Scalability: Building many stable qubits is difficult.
• Extreme cooling: Many systems require near absolute-zero temperatures.

Quantum computing is still in the NISQ (Noisy Intermediate-Scale Quantum) era but holds transformative potential.

Classical logic gates and quantum gates are both used to manipulate information, but they operate on fundamentally different principles.

Classical Logic Gates:

• Operate on classical bits (0 and 1).
• Examples: AND, OR, NOT, NAND, XOR.
• Generally irreversible (output does not always determine input, e.g., AND).
• Deterministic outputs.

Quantum Gates:

• Operate on qubits and their superpositions.
• Represented by unitary matrices and are always reversible.
• Preserve quantum information (no information loss).
• Can create superposition and entanglement.

Common Quantum Gates:

• Pauli-X Gate (NOT): Flips .
  

• Hadamard Gate (H): Creates superposition.
  

• CNOT Gate: Two-qubit gate used to create entanglement.

• Pauli-Z, Pauli-Y, Phase gates: Modify phase and rotation of qubit states.

Key Difference: Quantum gates are reversible and can exploit superposition and entanglement, giving quantum computers their unique power.

Modern computer architecture is evolving rapidly as traditional scaling laws reach their physical limits.

Background:

• Moore's Law: Predicted transistor count doubling roughly every two years — now slowing.
• Dennard Scaling: Predicted power density staying constant as transistors shrink — broke down around 2005, causing the power wall.

Latest Trends:

1. Multicore and Manycore Designs:

• Adding more cores instead of increasing clock speed to improve throughput.

2. Heterogeneous Computing:

• Combining CPUs, GPUs, and specialized accelerators (TPUs, NPUs, FPGAs) on the same system.

3. Domain-Specific Architectures (DSA):

• Custom hardware tailored for specific workloads (e.g., AI accelerators, tensor cores).

4. Chiplets and 3D Stacking:

• Building processors from smaller chiplets connected via advanced interconnects.
• 3D stacking increases density and reduces communication distance.

5. Near-Memory and In-Memory Computing:

• Reducing the memory wall by processing data closer to where it is stored.

6. Energy Efficiency Focus:

• Power efficiency (performance-per-watt) is now a primary design goal.

7. RISC-V and Open ISAs:

• Open-source instruction sets enabling customizable processors.

These trends collectively shift the focus from raw frequency scaling to specialization, parallelism, and efficiency.

Heterogeneous computing refers to systems that use more than one type of processor or core to handle different tasks efficiently, combining general-purpose and specialized processing units.

Components in a Heterogeneous System:

• CPU: Handles sequential control-intensive tasks.
• GPU: Handles massively parallel data workloads.
• NPU/TPU: Accelerates AI and neural network computations.
• DSP: Handles signal processing.
• FPGA: Provides reconfigurable custom logic.

Advantages:

• Improved Performance: Each task runs on the hardware best suited for it.
• Energy Efficiency: Specialized units perform tasks with less power.
• Scalability: Workloads can be distributed across diverse processors.
• Cost-effectiveness: Better performance-per-watt and per-dollar.

Examples:

• Smartphone SoCs (e.g., Apple A-series, Qualcomm Snapdragon) integrate CPU, GPU, NPU, ISP, and modem on one chip.
• AI servers combining CPUs with GPU/TPU accelerators.
• AMD APUs combining CPU and GPU on a single die.

Challenges:

• Complex programming models.
• Data movement and synchronization overhead between units.

Heterogeneous computing is central to modern architecture, powering everything from mobile devices to data centers.

Next-generation processor architecture focuses on overcoming the limitations of traditional monolithic chip design by adopting modular and advanced integration techniques.

1. Chiplet Architecture:

• Instead of a single large monolithic die, processors are built from multiple smaller dies called chiplets.
• Each chiplet can be manufactured on the most suitable process node.
• Advantages:
  • Higher manufacturing yield (smaller dies have fewer defects).
  • Modularity and reusability.
  • Lower cost.
• Example: AMD Ryzen/EPYC processors use chiplet designs.

2. 3D Stacking:

• Vertically stacking chips or memory using Through-Silicon Vias (TSVs).
• Reduces distance between components, lowering latency and power.
• Example: HBM (High Bandwidth Memory) stacked near the processor; AMD's 3D V-Cache.

3. Advanced Packaging:

• Technologies like 2.5D interposers and silicon bridges connect chiplets with high bandwidth.
• Enables integration of heterogeneous components in one package.

4. Other Features:

• Integrated accelerators (AI, security).
• Improved interconnects (e.g., Infinity Fabric, EMIB).
• Better power management and adaptive clocking.

These innovations allow continued performance scaling despite the slowdown of transistor miniaturization.

Microarchitecture (also called computer organization) refers to the way a given Instruction Set Architecture (ISA) is implemented in a particular processor. It describes the internal design and data paths that execute instructions.

ISA vs Microarchitecture:

Key Points:

• Multiple microarchitectures can implement the same ISA (e.g., Intel and AMD both implement x86 differently).
• The ISA remains stable for software compatibility, while microarchitecture evolves for performance.

Microarchitectural Elements:

• Pipeline stages (fetch, decode, execute, memory, write-back).
• Cache hierarchy (L1, L2, L3).
• Branch prediction and speculation.
• Out-of-order execution and superscalar units.

Thus, ISA defines what the processor does, while microarchitecture defines how it does it.

Modern processors employ several microarchitectural techniques to maximize instruction-level parallelism (ILP) and overall performance.

1. Pipelining:

• Divides instruction execution into stages (Fetch, Decode, Execute, Memory, Write-back).
• Multiple instructions overlap in different stages, improving throughput.
• Ideal speedup approaches the number of pipeline stages, but hazards limit it.

2. Superscalar Execution:

• The processor has multiple execution units and can issue more than one instruction per clock cycle.
• Increases ILP by executing independent instructions in parallel.

3. Out-of-Order (OoO) Execution:

• Instructions are executed based on data availability rather than program order.
• Uses techniques like register renaming and reservation stations to avoid stalls.
• Results are reordered before commit to maintain correctness.

4. Branch Prediction:

• Predicts the outcome of conditional branches to avoid pipeline stalls.
• Speculative execution proceeds along predicted paths.

5. Caching and Prefetching:

• Multi-level caches reduce memory latency.
• Prefetchers bring data into cache before it is needed.

6. SIMD Units:

• Perform the same operation on multiple data (vector processing).

Hazards Addressed:

• Structural, Data, and Control hazards are mitigated by forwarding, stalling, and prediction.

Together, these techniques enable modern CPUs to achieve high performance while maintaining ISA compatibility.

Tensor Cores are specialized processing units introduced in NVIDIA GPUs (starting with the Volta architecture) designed specifically to accelerate matrix operations that dominate deep learning and AI workloads.

Purpose:

• Deep learning relies heavily on matrix multiply-accumulate (MAC) operations, especially in neural network layers.
• Traditional CUDA cores handle these but less efficiently for large-scale AI.

How Tensor Cores Work:

• They perform a fused multiply-add on matrices in a single operation:
  
• Operate on small matrix tiles (e.g., ) in one clock cycle.
• Support mixed-precision computation (e.g., FP16 inputs with FP32 accumulation) for speed with accuracy.

Advantages:

• Massive throughput for matrix math compared to standard cores.
• Higher energy efficiency for AI training and inference.
• Support for reduced precision formats (FP16, BF16, INT8, TF32) enabling faster computation.

Applications:

• Training and inference of deep neural networks.
• Large language models and generative AI.
• Scientific computing with mixed precision.

Impact:

• Tensor Cores dramatically reduce training times, making them central to modern AI acceleration and NVIDIA's dominance in the AI hardware market.

Smartphone/tablet processors and desktop processors are designed with very different priorities.

Smartphone/Tablet Processors (SoCs):

• Based mostly on ARM architecture (RISC-based).
• Examples: Apple A17/M-series, Qualcomm Snapdragon, MediaTek Dimensity.
• Highly integrated System-on-Chip (SoC): CPU, GPU, NPU, ISP, modem on one die.
• Design goals: Power efficiency, thermal constraints, battery life.
• Use big.LITTLE heterogeneous core designs (high-performance + efficiency cores).
• Passively cooled (no fans).

Desktop Processors:

• Traditionally based on x86 architecture (Intel, AMD), though ARM is emerging.
• Examples: Intel Core i9, AMD Ryzen.
• Higher clock speeds and power budgets (often 65W–150W+).
• Design goals: Maximum performance, sustained heavy workloads.
• Actively cooled with fans/liquid cooling.
• Larger caches and more powerful cores.

Comparison Table:

Trend: The gap is narrowing as ARM-based chips (e.g., Apple M-series) now rival desktop performance while maintaining efficiency.

big.LITTLE is a heterogeneous processing architecture developed by ARM that combines two types of cores on the same chip to balance performance and power efficiency.

Core Types:

• big cores: High-performance cores designed for demanding tasks (gaming, video processing). They consume more power.
• LITTLE cores: Energy-efficient cores designed for light tasks (background processes, messaging). They consume much less power.

How It Works:

• The system dynamically assigns tasks to the appropriate cores.
• Light workloads run on LITTLE cores to save battery.
• Heavy workloads run on big cores for maximum performance.
• Modern designs use DynamIQ, allowing flexible clustering and even three tiers (prime, performance, efficiency cores).

Benefits for Smartphones:

• Extended battery life by using efficient cores for most tasks.
• High performance on demand without sacrificing efficiency.
• Better thermal management in fanless devices.
• Smooth user experience with responsive task scheduling.

Example:

• A processor might have 1 prime core + 3 performance cores + 4 efficiency cores.

This architecture is now standard in nearly all modern smartphone SoCs, enabling the balance of power and performance essential for mobile devices.

A System-on-Chip (SoC) is an integrated circuit that combines most or all components of a computer system onto a single chip.

Key Components of a Mobile SoC:

• CPU: Multiple cores (often big.LITTLE configuration).
• GPU: Handles graphics and parallel workloads.
• NPU (Neural Processing Unit): Accelerates AI/ML tasks.
• ISP (Image Signal Processor): Processes camera data.
• DSP (Digital Signal Processor): Handles audio and signal processing.
• Modem: Provides cellular/5G connectivity.
• Memory Controller & Cache: Manages RAM access.
• Connectivity blocks: Wi-Fi, Bluetooth, GPS.

Why SoCs Are Preferred in Mobile Devices:

• Space Efficiency: Integrating everything on one chip saves precious physical space.
• Power Efficiency: Shorter interconnects reduce power consumption and heat.
• Cost Reduction: Fewer separate components lower manufacturing cost.
• Performance: On-chip communication is faster than between separate chips.
• Reliability: Fewer external connections mean fewer points of failure.

Examples:

• Apple A17 Pro, Qualcomm Snapdragon 8 Gen series, Samsung Exynos.

SoCs are the foundation of modern smartphones, tablets, and increasingly laptops (e.g., Apple M-series), delivering high integration and efficiency.

SIMT (Single Instruction, Multiple Threads) is the execution model used by NVIDIA GPUs where a single instruction is executed by many threads in parallel, each operating on its own data.

SIMT Model:

• Threads are grouped into warps (typically 32 threads).
• All threads in a warp execute the same instruction at the same time but on different data.
• Each thread has its own registers and can follow independent control flow (though divergence reduces efficiency).

SIMD (Single Instruction, Multiple Data):

• A single instruction operates on multiple data elements packed into a vector register.
• Programmer/compiler must explicitly manage vector widths.
• Common in CPUs (e.g., SSE, AVX instructions).

Key Differences:

Branch Divergence in SIMT:

• When threads in a warp take different branches, execution is serialized, reducing performance.

Summary:

• SIMT offers a more flexible, thread-centric programming model, hiding vector complexity from the programmer, while SIMD requires explicit vectorization. SIMT is essentially SIMD with a thread abstraction layer.

GPGPU (General-Purpose computing on Graphics Processing Units) refers to using the GPU — originally designed for rendering graphics — to perform general computational tasks traditionally handled by the CPU.

Background:

• GPUs contain thousands of cores optimized for parallel arithmetic.
• Frameworks like CUDA (NVIDIA) and OpenCL (cross-platform) enable programmers to harness GPUs for non-graphics computation.

Why GPGPU Works:

• GPUs excel at performing the same operation on large amounts of data simultaneously (data parallelism).
• High memory bandwidth supports large data throughput.

Problems That Benefit Most:

• Data-parallel problems where the same operation applies to many elements:
  • Matrix and vector operations (linear algebra).
  • Image and video processing.
  • Deep learning training and inference.
  • Scientific simulations (physics, molecular dynamics).
  • Cryptography and hashing.
  • Financial modeling (Monte Carlo simulations).

Problems NOT Suited for GPGPU:

• Highly sequential tasks with heavy branching.
• Tasks with frequent, unpredictable control flow.
• Small workloads where data transfer overhead outweighs benefits.

Considerations:

• Data must be transferred between CPU (host) and GPU (device), which can be a bottleneck.
• Algorithms must be parallelizable to gain benefits.

GPGPU has transformed fields like AI, scientific computing, and data analytics by providing massive parallel processing power.

Building practical quantum computers faces major obstacles related to the fragile nature of quantum states, primarily decoherence and the need for quantum error correction.

1. Decoherence:

• Quantum states (superposition and entanglement) are extremely sensitive to their environment.
• Interaction with heat, electromagnetic noise, or vibrations causes qubits to lose their quantum properties — this is called decoherence.
• Coherence time (how long a qubit stays in a valid state) is very short, limiting computation time.
• Mitigation: Operating at temperatures near absolute zero and shielding qubits from noise.

2. Quantum Errors:

• Qubits are prone to errors such as bit-flip () and phase-flip errors.
• Unlike classical bits, qubits cannot be copied (No-Cloning Theorem), making error correction difficult.

3. Quantum Error Correction (QEC):

• Encodes a single logical qubit using many physical qubits to detect and correct errors.
• Example: The surface code may require hundreds or thousands of physical qubits per logical qubit.
• This creates enormous overhead in qubit requirements.

4. Other Challenges:

• Scalability: Adding more qubits while maintaining coherence and control is difficult.
• Gate fidelity: Quantum operations are imperfect and introduce errors.

Current State:

• We are in the NISQ (Noisy Intermediate-Scale Quantum) era — devices have limited qubits with significant noise.
• Achieving fault-tolerant quantum computing remains a key research goal.

Overcoming these challenges is essential before quantum computers can outperform classical machines on real-world problems.