Unit 2: Building Models with Keras - Subjective Questions

Keras is a high-level deep learning API used to build, train, evaluate, and deploy neural networks. It is included as the tf.keras module in TensorFlow.

Key features:

• User-friendly API: Models can be created using concise and readable Python code.
• Modularity: Layers, loss functions, optimizers, metrics, and callbacks can be combined as reusable components.
• Multiple model-building approaches: It supports the Sequential API, Functional API, and model subclassing.
• Hardware acceleration: Computations can run on CPUs, GPUs, and TPUs through TensorFlow.
• Built-in utilities: It provides tools for preprocessing data, tuning hyperparameters, saving models, and monitoring training.

Keras allows developers to focus on model architecture and experimentation while TensorFlow manages tensor operations, automatic differentiation, and hardware execution.

Keras is normally installed as part of TensorFlow.

Installation steps:

1. Create and activate a Python virtual environment.
2. Upgrade pip using python -m pip install --upgrade pip.
3. Install TensorFlow using pip install tensorflow.
4. Import Keras with from tensorflow import keras.

Verification:

• Run import tensorflow as tf.
• Display the version using print(tf.__version__).
• Check available GPU devices with tf.config.list_physical_devices("GPU").
• Build and summarize a small model to confirm that the Keras API is operational.

A compatible NVIDIA driver, CUDA environment, and cuDNN libraries may be required for GPU execution, depending on the TensorFlow version and installation method.

A Keras layer performs a tensor transformation and may contain trainable parameters such as weights and biases. Examples include Dense, Conv2D, LSTM, Embedding, Dropout, and BatchNormalization.

Model-building approaches:

• Sequential API: Represents a linear stack of layers. It is suitable when each layer has exactly one input and one output.
• Functional API: Represents a model as a graph of connected layers. It supports multiple inputs, multiple outputs, shared layers, and residual connections.
• Model subclassing: A custom class extends keras.Model and defines computation in its call method. It is appropriate for highly dynamic or nonstandard architectures.

The Sequential API is simplest, the Functional API offers flexible graph construction, and subclassing provides maximum control but usually requires more implementation effort.

A Keras regression workflow contains the following stages:

• Prepare the data: Separate input features and continuous targets, handle missing values, and divide the data into training, validation, and test sets.
• Normalize features: Fit normalization statistics only on the training data and apply the same transformation to all sets.
• Define the model: Use an input layer, one or more hidden Dense layers with nonlinear activations, and a final Dense(1) layer with a linear activation.
• Compile the model: Select an optimizer such as Adam, a regression loss such as mean squared error, and metrics such as MAE.
• Train the model: Call fit with training data, validation data, batch size, epochs, and suitable callbacks.
• Evaluate and predict: Use evaluate on unseen test data and predict to generate continuous outputs.

For targets and predictions , mean squared error is

Test data must remain excluded from preprocessing decisions and training to obtain an unbiased estimate of generalization.

A Multi-layer Perceptron (MLP) contains an input layer, one or more fully connected hidden layers, and an output layer.

During forward propagation, layer computes

followed by

where and are trainable parameters and is an activation function such as ReLU.

For multiclass classification, the output layer commonly uses softmax:

The predicted probabilities are compared with the true labels using cross-entropy loss. Backpropagation applies the chain rule to calculate the gradient of the loss with respect to each parameter. An optimizer then updates the parameters, for example:

where is the learning rate. Repeating this process over many batches and epochs enables the MLP to learn decision boundaries.

Binary classification:

• Each example belongs to one of two classes.
• The output is usually Dense(1, activation="sigmoid").
• The standard loss is binary_crossentropy.

Multiclass classification:

• Each example belongs to exactly one of classes.
• The output is usually Dense(K, activation="softmax").
• Use categorical_crossentropy for one-hot labels or sparse_categorical_crossentropy for integer labels.

Multilabel classification:

• Each example may belong to several classes simultaneously.
• The output is Dense(K, activation="sigmoid"), so each class receives an independent probability.
• The usual loss is binary_crossentropy.

Softmax probabilities compete and sum to one, whereas sigmoid outputs are independent. Therefore, softmax is appropriate for mutually exclusive classes, while sigmoid is appropriate for binary and multilabel decisions.

An image classification system can be developed through these stages:

• Load and split images: Create training, validation, and test datasets using class labels derived from folders or annotations.
• Preprocess data: Resize images to a fixed shape, scale pixel values, and batch the samples.
• Apply augmentation: Random flipping, rotation, translation, zooming, or contrast changes can improve generalization.
• Construct a CNN: Use repeated Conv2D and pooling layers to extract spatial features, followed by global pooling or flattening and a classification layer.
• Compile: Choose a suitable cross-entropy loss, an optimizer such as Adam, and metrics such as accuracy.
• Train: Monitor validation performance and use callbacks such as early stopping and model checkpointing.
• Evaluate: Measure test accuracy and inspect a confusion matrix, precision, recall, and class-specific errors.

Convolutional layers learn local patterns such as edges and textures. Deeper layers combine these patterns into higher-level object features. For limited datasets, transfer learning with a pretrained network is often more effective than training a CNN entirely from scratch.

Training from scratch initializes every model parameter randomly and learns all visual features from the target dataset. It provides complete architectural control but generally requires a large labeled dataset, substantial computation, and careful regularization.

Transfer learning begins with a model pretrained on a large dataset. A common procedure is:

• Load a pretrained base without its original classifier.
• Freeze the base model.
• Add task-specific pooling and classification layers.
• Train the new classifier.
• Optionally unfreeze selected upper layers and fine-tune them using a small learning rate.

Transfer learning usually converges faster and performs better when labeled data is limited. However, its benefit may decrease when the source images differ greatly from the target domain. Fine-tuning must be controlled because large parameter updates can destroy useful pretrained representations.

A text classification pipeline includes the following steps:

• Collect and label text: Prepare documents or sentences with their target classes.
• Split the dataset: Create training, validation, and test sets before adapting preprocessing components.
• Standardize text: Optionally convert case and remove or normalize selected punctuation.
• Tokenize and vectorize: Use a TextVectorization layer to map tokens to integer IDs and pad or truncate sequences to a fixed length.
• Learn representations: Pass token IDs through an Embedding layer.
• Model the sequence: Use global pooling, a CNN, an RNN such as LSTM or GRU, or a transformer-based encoder.
• Classify: Use sigmoid for binary or multilabel output and softmax for mutually exclusive multiclass output.
• Train and evaluate: Select the matching loss function and inspect metrics beyond accuracy when classes are imbalanced.

The vocabulary must be adapted only on training text to avoid data leakage. Unknown words should map to an out-of-vocabulary token so that unseen text can still be processed.

Tokenization divides text into units such as words or subwords and maps each unit to an integer index. A fixed vocabulary determines which tokens receive dedicated indices.

Sequence padding and truncation create tensors of consistent length so that multiple sequences can be processed in one batch. Short sequences are padded, while sequences longer than the chosen limit are truncated.

An embedding layer maps each token index to a dense vector. If the vocabulary size is and the embedding dimension is , the layer learns an embedding matrix

The row is the representation of token . Unlike one-hot vectors, embeddings are compact and trainable, allowing tokens used in similar contexts to develop similar representations. Padding positions should be masked when the selected sequence model supports masking.

Underfitting occurs when a model cannot capture important patterns in the training data. Both training and validation losses remain high, and both performances are poor. Causes include insufficient model capacity, inadequate training, excessive regularization, or unsuitable features.

Overfitting occurs when a model learns training-specific details that do not generalize. Training loss continues to decrease while validation loss stops improving or increases. The resulting gap between training and validation performance is called the generalization gap.

Diagnosis:

• High training and validation error indicates underfitting.
• Low training error but much higher validation error indicates overfitting.
• Similar and acceptably low training and validation errors indicate a better bias-variance balance.

Learning curves should be interpreted over multiple epochs, and the final conclusion should be confirmed on an untouched test set.

Important methods for reducing overfitting include:

• More representative data: Additional diverse samples reduce reliance on accidental patterns.
• Data augmentation: Label-preserving transformations generate varied training inputs.
• Dropout: Randomly sets a fraction of activations to zero during training.
• Weight regularization: L1 or L2 penalties discourage unnecessarily large weights. An L2-regularized objective can be written as

• Early stopping: Stops training when validation performance no longer improves and can restore the best weights.
• Reduced capacity: Fewer layers or units can prevent memorization.
• Transfer learning: Pretrained representations can improve generalization on smaller datasets.
• Normalization and proper validation: Stable inputs and a representative validation set support reliable model selection.

Regularization strength should be selected using validation data. Excessive regularization can produce underfitting.

Keras supports saving either a complete model or only its weights.

Complete-model saving:

• model.save("classifier.keras") stores the architecture, learned weights, and available training configuration.
• keras.models.load_model("classifier.keras") reconstructs the saved model.

Weights-only saving:

• model.save_weights("model.weights.h5") stores parameter values.
• To restore them, the same compatible architecture must first be created and then load_weights is called.

Saving the complete model is useful because it preserves the model structure and state needed for inference or continued training without manually rebuilding the architecture. A model may also be exported in a deployment-oriented format when it must be served outside the original training program.

After loading, developers should verify predictions using known samples and ensure that preprocessing, custom layers, and software versions remain compatible.

Keras callbacks are objects that execute actions at selected stages of training, such as the beginning or end of an epoch.

• EarlyStopping: Monitors a quantity such as validation loss and stops training after it fails to improve for a specified patience period. restore_best_weights=True restores the best observed weights.
• ModelCheckpoint: Saves the model or weights during training. With save_best_only=True, it retains the checkpoint with the best monitored score.
• ReduceLROnPlateau: Reduces the learning rate when a monitored metric stops improving, allowing smaller optimization steps near a possible minimum.

These callbacks improve training efficiency and reliability. Their monitor, mode, patience, and filename settings must be configured consistently with the selected objective and metric.

Hyperparameters are configuration values chosen before or around training rather than directly learned through backpropagation. Examples include the number of layers, units per layer, activation function, learning rate, batch size, dropout rate, and regularization strength.

A systematic tuning procedure is:

1. Define a search space and a function that constructs the model from candidate values.
2. Select an objective such as validation loss or validation accuracy.
3. Choose a search strategy such as random search, Bayesian optimization, or Hyperband.
4. Train each trial under comparable data splits, epoch limits, and stopping rules.
5. Select the configuration with the strongest validation performance.
6. Retrain the selected configuration as appropriate and evaluate it once on the untouched test set.

KerasTuner can automate trial generation and tracking. The test set must not guide hyperparameter choices because repeated test evaluation causes optimistic and biased performance estimates.

Grid search evaluates every predefined combination. It is simple and reproducible but becomes expensive as the number of hyperparameters grows.

Random search samples combinations from specified distributions. It often explores large spaces more efficiently because it does not spend equal effort on every dimension.

Bayesian optimization uses results from previous trials to estimate which configurations are promising. It can reduce the number of expensive evaluations but adds algorithmic complexity.

Hyperband allocates small training budgets to many candidates and progressively assigns more resources to the best-performing ones. It is effective when poor configurations can be identified early.

The preferred method depends on search-space size, training cost, available parallel hardware, and the reliability of early validation measurements. Random search or Hyperband is generally more practical than exhaustive grid search for deep neural networks.

The NVIDIA DGX Station A100 is an integrated, data-center-class AI workstation intended for deskside use. It combines NVIDIA A100 Tensor Core GPUs with a high-performance CPU platform, large system and GPU memory, fast storage, high-bandwidth GPU interconnects, and an optimized AI software environment.

Importance for AI development:

• It supports training, fine-tuning, inference, and data-science workloads on one system.
• Tensor Cores accelerate mixed-precision matrix operations used by deep learning.
• Multiple GPUs enable larger models and distributed computation.
• Large GPU memory supports demanding models and datasets.
• The integrated software stack reduces environment setup and compatibility work.
• It allows teams to prototype locally while following software practices similar to larger NVIDIA DGX infrastructure.

It is particularly useful for researchers and development teams that require substantial AI compute without relying exclusively on a remote data center.

The DGX Station A100 hardware architecture is organized around multiple NVIDIA A100 Tensor Core GPUs connected through high-bandwidth GPU interconnect technology.

Major components and roles:

• A100 GPUs: Execute massively parallel tensor and matrix computations used in training and inference.
• Tensor Cores: Accelerate mixed-precision operations, including formats designed for high-throughput AI computation.
• Large GPU memory: Stores model parameters, activations, and larger batches, reducing transfers to system memory.
• NVLink and NVSwitch-class connectivity: Enables high-bandwidth, low-latency communication among GPUs for distributed training.
• Server-grade CPU and system memory: Handle data preparation, orchestration, storage operations, and CPU-based tasks.
• High-speed NVMe storage: Improves dataset loading, checkpointing, and experiment management.
• Cooling and power subsystems: Sustain intensive computation in a workstation form factor.

Together, these components reduce compute and communication bottlenecks. Actual component capacities can vary by system configuration, so workload planning should use the specifications of the installed model.

The DGX Station A100 software stack integrates the operating environment, NVIDIA drivers, GPU computing libraries, management tools, containers, and optimized AI frameworks.

Typical stack components:

• DGX operating environment: Provides a supported Linux-based platform and system configuration.
• NVIDIA drivers: Allow the operating system and applications to communicate with the GPUs.
• CUDA and accelerated libraries: Supply GPU programming and optimized mathematical operations.
• Framework containers: Provide tested versions of TensorFlow, PyTorch, and related dependencies.
• NGC catalog: Distributes optimized containers, pretrained models, and development resources.
• Monitoring and management tools: Report GPU utilization, memory consumption, temperature, and process activity.

Containers isolate application dependencies and make experiments more reproducible. They also simplify moving a workload between compatible DGX systems or cloud infrastructure. The host driver must remain compatible with the CUDA runtime expected by the selected container.

The CUDA Toolkit is NVIDIA's development platform for creating GPU-accelerated applications.

Important components include:

• The nvcc CUDA compiler and development tools.
• Runtime and driver-facing APIs for launching GPU kernels and managing memory.
• Optimized libraries such as cuBLAS for linear algebra and cuFFT for Fourier transforms.
• Profiling and debugging tools for identifying compute, memory, and communication bottlenecks.
• Integration points used by deep learning libraries and frameworks.

Keras developers usually access CUDA indirectly through TensorFlow, which maps supported operations to GPU kernels and libraries such as cuDNN.

Systems such as the DGX Station A100 can advance AI by shortening experimentation cycles, supporting larger and more capable models, enabling mixed-precision and multi-GPU training, and providing a consistent path from local prototypes to scalable infrastructure. Future progress will also depend on energy efficiency, memory capacity, interconnect speed, optimized software, responsible model development, and wider access to computational resources. Hardware acceleration improves feasibility, but dataset quality, algorithm design, security, fairness, and governance remain essential.