Unit 2: Building Models with Keras

I. Orientation — Model Building in Deep Learning

Deep learning models learn hierarchical representations by passing data through layers of parameterized transformations. Keras, created by François Chollet and first released in 2015, provides a high-level API for defining, training, evaluating, and deploying such models with comparatively little code.

• Learning principle: Training adjusts weights (W) and biases (b) to minimize a loss function (L) on observed data.
• Basic transformation: A dense layer computes:
  [
  \mathbf{y}=f(\mathbf{W}\mathbf{x}+\mathbf{b})
  ]
  where (\mathbf{x}) is the input, (\mathbf{W}) contains weights, (\mathbf{b}) contains biases, and (f) is an activation function.
• Training cycle:
  1. Forward propagation produces predictions.
  2. A loss function measures prediction error.
  3. Backpropagation calculates gradients.
  4. An optimizer updates parameters.
• Data convention: Data is commonly divided into training, validation, and test sets.
• Core objective: A useful model must generalize to unseen data rather than merely memorize its training examples.

II. Keras Foundations — APIs, Installation, Layers, and Models

A. Introduction to Keras

Keras is a high-level deep learning API designed around simplicity, modularity, and rapid experimentation.

• Backend support: Keras 3 can operate with TensorFlow, JAX, or PyTorch as its computational backend.
• Unified workflow: The typical process uses model.compile(), model.fit(), model.evaluate(), and model.predict().
• Main advantages:
  • Readable syntax: Models are assembled from reusable layer objects.
  • Automatic differentiation: The backend calculates gradients required by backpropagation.
  • Hardware acceleration: Operations can run on CPUs, GPUs, or specialized accelerators.
• Main APIs:
  1. Sequential API: Suitable for a linear stack of layers.
  2. Functional API: Supports multiple inputs, multiple outputs, and branching graphs.
  3. Model subclassing: Provides maximum control by defining custom forward computation.

B. Keras installation

Keras is installed as a Python package and configured with a supported backend.

• Basic installation:
  
  BASH

    python -m pip install keras tensorflow

• Backend selection: The backend must be selected before importing Keras.
  
  PYTHON

    import os
    os.environ["KERAS_BACKEND"] = "tensorflow"
    import keras
    print(keras.__version__)

• Environment practice: A virtual environment prevents package conflicts.
  
  BASH

    python -m venv dl_env

• GPU verification: With TensorFlow, tf.config.list_physical_devices("GPU") lists detected GPUs.
• Compatibility requirement: Python, GPU drivers, backend libraries, and CUDA-related components must have compatible versions.

C. Keras layers and models

Layers perform transformations, while a model connects layers into a trainable computational graph.

• Common layers:
  • Dense: Fully connects inputs to output units.
  • Conv2D: Extracts local spatial features from images.
  • MaxPooling2D: Reduces spatial dimensions.
  • Embedding: Converts integer token IDs into dense vectors.
  • Dropout: Randomly removes activations during training.
• Sequential example:
  
  PYTHON

    from keras import Sequential, layers
  
    model = Sequential([
        layers.Input(shape=(20,)),
        layers.Dense(64, activation="relu"),
        layers.Dense(1)
    ])

• Model state: Trainable parameters include layer weights and biases; non-trainable state can include normalization statistics.
• Shape rule: The output shape of one layer must be compatible with the input shape expected by the next.

III. Supervised Models — Regression and Classification

A. Building a regression model

A regression model predicts a continuous quantity such as price, temperature, or demand.

• Output design: A single linear output neuron is appropriate for one unrestricted numerical target.
• Loss function: Mean squared error is:
  [
  \operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2
  ]
  where (n) is the number of samples, (y_i) is the true value, and (\hat y_i) is the prediction.
• Implementation:
  
  PYTHON

    model.compile(optimizer="adam", loss="mse", metrics=["mae"])
    history = model.fit(x_train, y_train,
                        validation_split=0.2, epochs=50)

• Preprocessing: Standardization, (z=(x-\mu)/\sigma), prevents large-scale features from dominating optimization.
• Evaluation: Mean absolute error reports error in the target’s original unit and is less sensitive to extreme errors than MSE.

B. Multi-layer Perceptron learning for classification

A multi-layer perceptron learns nonlinear class boundaries through one or more fully connected hidden layers.

• Hidden computation: ReLU commonly applies (f(z)=\max(0,z)), enabling nonlinear feature combinations.
• Binary classification: One sigmoid output represents:
  [
  p(y=1\mid\mathbf{x})=\frac{1}{1+e^{-z}}
  ]
• Multiclass classification: A softmax output contains one unit per class:
  [
  p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}
  ]
  where (p_i) is the predicted probability of class (i).
• Loss pairing:
  1. Binary output: Sigmoid with binary cross-entropy.
  2. Multiclass output: Softmax with categorical or sparse categorical cross-entropy.
• Decision rule: argmax selects the highest-probability class in multiclass prediction.

C. Image classification with Keras

Image classification assigns an input image to a predefined category, usually using convolutional neural networks.

• Input representation: A color image normally has shape (height, width, 3), where three channels represent red, green, and blue.
• Feature extraction: Early convolutional filters detect edges; deeper filters combine them into textures, parts, and objects.
• Basic architecture:
  
  PYTHON

    model = Sequential([
        layers.Input(shape=(128, 128, 3)),
        layers.Rescaling(1./255),
        layers.Conv2D(32, 3, activation="relu"),
        layers.MaxPooling2D(),
        layers.Conv2D(64, 3, activation="relu"),
        layers.GlobalAveragePooling2D(),
        layers.Dense(10, activation="softmax")
    ])

• Augmentation: Random flips, rotations, or zooms create plausible variations and improve generalization.
• Transfer learning: Pretrained networks such as ResNet or EfficientNet provide reusable visual features when labeled data is limited.

D. Building a text classification model

A text classification model predicts categories such as sentiment, topic, intent, or spam status from textual input.

• Vectorization: TextVectorization standardizes text, creates a vocabulary, and maps tokens to integer sequences.
• Embedding: An embedding matrix maps each token ID to a learned vector of dimension (d).
• Architecture choices: Pooled embeddings provide a simple baseline, while CNNs, recurrent networks, and Transformers model richer contextual patterns.
• Example pipeline:
  
  PYTHON

    model = Sequential([
        layers.Embedding(vocabulary_size, 128),
        layers.GlobalAveragePooling1D(),
        layers.Dense(64, activation="relu"),
        layers.Dense(1, activation="sigmoid")
    ])

• Sequence control: Padding creates equal-length batches, while truncation limits memory use.
• Data integrity: Vocabulary adaptation must use training text only to avoid leaking information from validation or test data.

IV. Model Reliability and Lifecycle — Generalization, Persistence, and Optimization

A. Overfitting and underfitting

Overfitting and underfitting describe failures to achieve an appropriate balance between model capacity and generalization.

1. Underfitting:
   • Pattern: Training and validation losses both remain high.
   • Cause: The model may be too simple, insufficiently trained, or supplied with weak features.
   • Remedy: Increase capacity, improve features, or train longer.
2. Overfitting:
   • Pattern: Training loss falls while validation loss begins rising.
   • Cause: The model learns training-specific noise.
   • Remedy: Use more data, augmentation, dropout, weight regularization, or early stopping.
   • Regularization: L2 regularization adds (\lambda\sum_j w_j^2) to the loss, where (\lambda) controls penalty strength.
   • Monitoring: Validation curves reveal the epoch at which generalization starts deteriorating.

B. Saving and loading a model

Model persistence stores learned parameters and configuration so that training or inference can continue later.

• Complete model:
  
  PYTHON

    model.save("classifier.keras")
    restored_model = keras.models.load_model("classifier.keras")

• Stored information: The .keras format can preserve architecture, weights, compilation configuration, and optimizer state.
• Weights only:
  
  PYTHON

    model.save_weights("weights.weights.h5")
    model.load_weights("weights.weights.h5")

• Requirement: Loading weights alone requires a compatible architecture with matching parameter shapes.
• Operational practice: Save preprocessing rules, class labels, software versions, and input signatures alongside the model.

C. Hyperparameter tuning

Hyperparameter tuning searches for settings chosen before training rather than learned through gradient descent.

• Typical hyperparameters: Learning rate, batch size, layer count, units, dropout rate, activation, optimizer, and convolutional filter size.
• Search methods:
  1. Grid search: Tests every specified combination but becomes expensive rapidly.
  2. Random search: Samples combinations and often explores large spaces more efficiently.
  3. Bayesian optimization: Uses previous results to select promising trials.
• KerasTuner pattern:
  
  PYTHON

    units = hp.Int("units", 32, 256, step=32)
    rate = hp.Float("dropout", 0.0, 0.5, step=0.1)

• Selection rule: Compare trials using a validation metric, then evaluate the chosen configuration once on untouched test data.
• Resource control: Early stopping and limited trial budgets reduce computation.

V. NVIDIA DGX Station A100 — Accelerated AI Infrastructure

A. NVIDIA DGX Station A100

NVIDIA DGX Station A100 is an integrated workstation-class system intended for computationally intensive AI development.

• Primary role: It brings data-center-style multi-GPU computing to an office or laboratory environment.
• Core accelerator: NVIDIA A100 Tensor Core GPUs accelerate matrix operations central to neural-network training.
• Workload range: The system supports model development, distributed training, inference, analytics, and scientific computing.
• Integrated design: Compute hardware, drivers, communication libraries, and optimized frameworks are validated as one platform.
• Practical benefit: Local processing can reduce dependence on shared clusters and help keep sensitive datasets on premises.

B. Introduction to NVIDIA DGX Station A100

The DGX Station A100 provides a preconfigured environment for researchers who need substantial AI performance without operating a conventional server room.

• User focus: Data scientists can move from prototyping to large experiments on the same local platform.
• Multi-user operation: GPU resources can be shared among several users or assigned to separate workloads.
• A100 features: Tensor Cores support accelerated numerical formats, while Multi-Instance GPU technology can partition supported GPU resources into isolated instances.
• Deployment context: Typical uses include computer vision, natural-language processing, recommender systems, and simulation.
• Constraint: High acquisition cost, power demand, and specialized administration make the platform unsuitable for lightweight workloads.

C. Hardware Architecture

The hardware architecture combines multiple high-memory GPUs with a server-grade processor, storage, memory, and high-speed interconnection.

• GPU subsystem: Four A100 GPUs provide large aggregate accelerator memory for datasets, activations, and model parameters.
• GPU communication: NVIDIA NVLink enables faster GPU-to-GPU data exchange than routing all communication through ordinary CPU pathways.
• CPU subsystem: An AMD EPYC processor handles operating-system tasks, preprocessing, storage operations, and workload coordination.
• System memory: Large-capacity ECC memory supports data pipelines and helps detect or correct memory errors.
• Storage: NVMe solid-state storage supplies high throughput for datasets and checkpoints.
• Cooling: A self-contained cooling design permits workstation deployment while controlling heat from sustained multi-GPU workloads.

D. DGX Station A100 Software Stack

The DGX software stack supplies validated system software and optimized AI libraries above the physical hardware.

• Operating environment: NVIDIA DGX OS is based on Ubuntu Linux and includes platform-specific configuration.
• Drivers and runtime: NVIDIA drivers expose GPUs to applications, while CUDA provides the parallel-computing environment.
• Libraries: cuDNN accelerates deep neural-network primitives; NCCL supports communication operations such as all-reduce across GPUs.
• Containers: NVIDIA Container Toolkit allows containers to access GPUs, improving portability and dependency isolation.
• Framework access: Optimized containers for TensorFlow, PyTorch, and related tools are distributed through the NVIDIA NGC catalog.
• Management value: Validated updates reduce configuration errors compared with assembling an equivalent stack manually.

E. CUDA Toolkit

The CUDA Toolkit is NVIDIA’s development platform for executing general-purpose parallel programs on compatible GPUs.

• Programming model: A CPU host launches GPU kernels, and thousands of lightweight threads process data concurrently.
• Hierarchy: Threads are organized into blocks, and blocks form a grid.
• Core components:
  • Compiler: nvcc compiles CUDA C/C++ code.
  • Runtime: Manages devices, kernels, and memory transfers.
  • Libraries: cuBLAS accelerates linear algebra, while cuFFT performs fast Fourier transforms.
• Deep learning connection: Keras normally accesses CUDA indirectly through a backend such as TensorFlow.
• Performance condition: Efficient code minimizes host-device transfers and uses parallelism large enough to offset kernel-launch overhead.

F. Future of AI with DGX Station A100

DGX-class systems illustrate the movement toward larger, faster, and more locally accessible AI computing platforms.

• Larger models: High-memory multi-GPU systems support increasingly parameter-intensive generative and multimodal models.
• Mixed precision: Lower-precision formats increase throughput and reduce memory consumption while preserving acceptable numerical accuracy.
• Local AI: On-premises computation supports privacy-sensitive healthcare, finance, engineering, and government workloads.
• Scalable development: Containerized software allows experiments developed locally to migrate to larger DGX clusters or cloud infrastructure.
• Continuing limitation: Energy use, hardware cost, model bias, security, and environmental impact remain important considerations as computing capacity expands.