Unit 1: Building Models with TensorFlow - Subjective Questions

TensorFlow is an open-source numerical computation and machine learning framework developed by Google. It represents data as tensors and provides tools for constructing, training, evaluating, and deploying models.

Major features:

• Tensor operations: Efficiently performs mathematical operations on multidimensional arrays.
• Automatic differentiation: Uses tf.GradientTape to calculate gradients required for training.
• Hardware acceleration: Executes operations on CPUs, GPUs, and TPUs.
• Keras integration: Provides the high-level tf.keras API for rapid model development.
• Scalable deployment: Supports mobile, browser, server, and cloud environments.
• Visualization: Integrates with TensorBoard to inspect metrics, graphs, and model behavior.
• Distributed training: Can train large models across multiple devices or machines.

TensorFlow is normally installed in an isolated Python environment to avoid dependency conflicts.

Installation steps:

1. Install a supported version of Python and pip.
2. Create a virtual environment using python -m venv tf-env.
3. Activate the environment.
4. Upgrade pip using python -m pip install --upgrade pip.
5. Install TensorFlow using pip install tensorflow.
6. For hardware acceleration, install the platform-specific GPU drivers and dependencies supported by the selected TensorFlow version.

Verification:

• Import TensorFlow with import tensorflow as tf.
• Display the version using print(tf.__version__).
• Create a tensor, such as tf.constant([1, 2, 3]).
• Check available GPUs using tf.config.list_physical_devices("GPU").

A successful import and tensor operation confirm that the core installation works.

A tensor is a multidimensional data structure used by TensorFlow to represent input data, parameters, and computation results. Its rank is the number of axes or dimensions.

• Scalar: Rank 0, such as tf.constant(7). It has no axis.
• Vector: Rank 1, such as tf.constant([2, 4, 6]). It has one axis.
• Matrix: Rank 2, such as tf.constant([[1, 2], [3, 4]]). It has rows and columns.
• Higher-rank tensor: Rank 3 or more. For example, a batch of RGB images may have shape (batch, height, width, channels) and rank 4.

For a tensor , the rank is 4 and the shape records the size of each axis. Rank describes the number of axes, whereas shape describes their lengths.

The structural properties of a tensor describe different aspects of its organization:

• Rank: Number of axes. A tensor with shape (3, 4, 5) has rank 3. It can be obtained with tf.rank(tensor).
• Shape: Number of elements along each axis. It can be inspected through tensor.shape or tf.shape(tensor).
• Size: Total number of elements. For shape , the size is . TensorFlow provides tf.size(tensor).
• Data type: Type of each stored value, such as tf.float32, tf.int32, or tf.bool. It is available as tensor.dtype.

Two tensors may have the same size but different shapes and ranks. For example, shapes (12,) and (3, 4) both contain 12 elements but have ranks 1 and 2 respectively.

A computation graph represents a computation as connected operations and tensors. Operations form the nodes, while tensors flowing between operations form the edges. Graphs allow TensorFlow to analyze, optimize, serialize, and execute computations efficiently.

Eager execution:

• Operations run immediately when called.
• Results are directly available as Python values or tensors.
• Debugging is straightforward.
• It is the default mode in TensorFlow 2.

Graph execution:

• TensorFlow first traces Python code and builds a graph, commonly through @tf.function.
• The graph can be optimized and executed efficiently.
• It improves portability and may improve performance.
• Python side effects and dynamic Python behavior require care during tracing.

Thus, eager mode is convenient for development, while graph mode is useful for optimized training and deployment.

The @tf.function decorator converts a Python function containing TensorFlow operations into a callable graph representation. When the function is first invoked, TensorFlow executes a process called tracing.

During tracing, TensorFlow observes tensor operations, creates a graph, and stores a concrete function specialized for the input signature. Later calls with compatible inputs reuse that graph.

Retracing occurs when TensorFlow creates another graph, often because input shapes, data types, or Python argument values change. Excessive retracing is undesirable because it:

• Adds graph-construction overhead.
• Increases memory consumption.
• Can reduce overall performance.

Retracing can be reduced by supplying a stable input_signature, passing tensors instead of frequently changing Python values, and keeping input shapes and data types consistent.

A TensorFlow variable is a mutable tensor-like object used to store persistent state, especially model weights and biases. It is created using tf.Variable.

Comparison:

• tf.Variable is mutable; tf.constant is immutable.
• Variables maintain state across repeated function calls.
• Trainable variables are automatically tracked by Keras layers and models.
• Constants are suitable for fixed values that do not change during computation.

Assignment operations:

• assign(value) replaces the current value.
• assign_add(value) adds to the current value.
• assign_sub(value) subtracts from the current value.

For example, if and w.assign_sub(0.1 * gradient) is executed with gradient , the new value becomes . Variables must generally retain compatible shapes and data types during assignment.

TensorFlow uses automatic differentiation to calculate derivatives of operations recorded during a forward computation. Within a tf.GradientTape context, TensorFlow records operations involving watched tensors and trainable variables. The method tape.gradient(target, source) then applies the chain rule.

Given

the derivative is

At ,

A TensorFlow implementation creates x = tf.Variable(2.0), computes y inside with tf.GradientTape() as tape:, and obtains the derivative through tape.gradient(y, x). Persistent tapes can calculate multiple gradients, but they consume more resources and should be released after use.

An optimizer updates trainable model parameters to minimize a loss function. During training, TensorFlow computes the loss, differentiates it with respect to the parameters, and asks the optimizer to apply the resulting gradients.

For a parameter , gradient descent uses

where:

• is the loss measuring prediction error.
• is the gradient indicating the direction of greatest increase.
• is the learning rate controlling update size.

A very large learning rate may overshoot or cause divergence, while a very small value may make training slow. In TensorFlow, optimizer.apply_gradients(...) applies computed gradients, while Keras model.fit(...) performs this training process automatically after the model is compiled.

These optimizers differ in how they transform gradients into parameter updates.

• SGD: Applies updates directly from the current gradient. It is simple and memory-efficient but can oscillate or converge slowly.
• Momentum: Accumulates a moving direction from previous gradients, accelerating progress along consistent directions and reducing oscillation.
• RMSprop: Divides the gradient by a moving average of squared gradients, giving each parameter an adaptive effective learning rate.
• Adam: Combines momentum-like first-moment estimates with RMSprop-like second-moment estimates and includes bias correction.

Adam is often a strong default for rapid experimentation, while SGD with momentum may provide strong generalization in some tasks. The best choice depends on the model, data, batch size, and learning-rate schedule; no optimizer is universally superior.

TensorFlow provides several operations for changing tensor organization:

• tf.reshape(t, new_shape) rearranges elements into a new shape without changing their values. The total number of elements must remain unchanged. A dimension of -1 may be inferred.
• tf.transpose(t, perm) reorders axes according to the given permutation. For a matrix, it can exchange rows and columns.
• tf.expand_dims(t, axis) inserts an axis of length 1, such as converting shape (h, w) to (1, h, w).
• tf.squeeze(t, axis) removes axes whose length is 1.

Reshaping does not itself permute elements according to semantic axes, whereas transposition explicitly changes axis order. Squeezing an axis whose length is not 1 is invalid.

These operations select or combine multidimensional data.

• Indexing: t[0] selects the first item along the first axis.
• Slicing: t[:, 1:3] selects all rows and columns with indices 1 and 2.
• Concatenation: tf.concat([a, b], axis=0) joins tensors along an existing axis. All dimensions other than the concatenation axis must be compatible.
• Stacking: tf.stack([a, b], axis=0) creates a new axis and requires all input tensors to have the same shape.

If a and b each have shape (2, 3), concatenation on axis 0 produces (4, 3), while stacking on axis 0 produces (2, 2, 3). Therefore, concatenation extends an existing dimension, whereas stacking increases the rank.

Broadcasting allows TensorFlow to perform element-wise operations on tensors with different but compatible shapes without explicitly copying data. Dimensions are compared from right to left. Two dimensions are compatible when they are equal or one of them is 1. Missing leading dimensions are treated as 1.

For shapes and :

• Rightmost dimensions and broadcast to .
• Middle dimensions and broadcast to .
• Leftmost dimensions and broadcast to .

Therefore, the result has shape

Broadcasting is useful for adding a bias vector to a batch of activations. However, incompatible dimensions, such as 3 and 4 at the same aligned position, cause an error.

Element-wise multiplication uses tf.multiply(a, b) or a * b and multiplies corresponding elements, subject to broadcasting. Matrix multiplication uses tf.matmul(a, b) or a @ b and computes dot products between rows and columns.

For matrices and ,

with each element

For batched tensors with shapes (..., m, n) and (..., n, p), TensorFlow broadcasts compatible leading batch dimensions and returns shape (..., m, p). The inner dimensions must match unless one operand is explicitly transposed through transpose_a or transpose_b.

TensorBoard is TensorFlow's visualization toolkit for inspecting experiments and model behavior. It can display losses, metrics, computation graphs, histograms, images, embeddings, and profiling information.

Typical Keras workflow:

1. Create a log directory for the experiment.
2. Construct tf.keras.callbacks.TensorBoard(log_dir=...).
3. Pass the callback to model.fit(...).
4. Start TensorBoard with tensorboard --logdir <directory>.
5. Open the displayed local address in a browser.

For custom loops, tf.summary.create_file_writer(...) creates a writer, and functions such as tf.summary.scalar(...) record values at specified steps. Separate log directories should be used for different runs so that hyperparameters and learning curves can be compared reliably.

TensorBoard helps diagnose training by plotting training and validation metrics over time.

• Underfitting: Both training and validation losses remain high, suggesting insufficient model capacity, weak features, excessive regularization, or inadequate training.
• Overfitting: Training loss continues decreasing while validation loss rises or stops improving. Possible responses include regularization, dropout, data augmentation, early stopping, or more data.
• Optimization instability: Strong oscillation, sudden spikes, or NaN values may indicate an excessive learning rate, exploding gradients, invalid data, or numerical instability.
• Slow convergence: Nearly flat curves may indicate a learning rate that is too low, poor initialization, saturated activations, or unsuitable optimization.
• Parameter inspection: Histograms reveal changing weight and activation distributions.

TensorBoard provides evidence, but the curves must be interpreted alongside the dataset, model architecture, and experimental settings.

Deep learning is a branch of machine learning that uses neural networks with multiple representation-learning layers. Each layer transforms its input, allowing the model to learn increasingly abstract features.

Differences from traditional machine learning:

• Deep learning often learns features directly from raw data; traditional methods commonly depend more on manual feature engineering.
• Deep models generally require more data and computational power.
• They are especially effective for unstructured data such as images, speech, and text.
• Traditional models can be easier to train and interpret for small structured datasets.
• Deep neural networks are trained end to end using backpropagation and gradient-based optimization.

Deep learning is not automatically better for every task. Dataset size, latency, interpretability, hardware, and maintenance constraints should determine the approach.

A feedforward neural network contains an input layer, one or more hidden layers, and an output layer. Information flows from inputs toward outputs without recurrent connections.

For one layer, the pre-activation and output are


where contains weights, is the bias, and is an activation function such as ReLU. Multiple layers compose these transformations to learn nonlinear relationships.

The network's prediction is compared with the target using a loss function. Backpropagation applies the chain rule to calculate each parameter's contribution to the loss. For composed functions,

An optimizer then updates the parameters. Repeating forward propagation, loss calculation, backpropagation, and updating over many batches trains the network.

A complete TensorFlow workflow includes the following stages:

1. Prepare data: Load, clean, normalize, split, batch, and optionally shuffle the data, often with tf.data.
2. Define the model: Create layers using tf.keras.Sequential or the Functional API.
3. Compile: Select an optimizer, a suitable loss function, and evaluation metrics.
4. Train: Call model.fit(...) with training data, validation data, epochs, and callbacks.
5. Monitor: Inspect loss and metrics through returned history and TensorBoard.
6. Evaluate: Use model.evaluate(...) on unseen test data.
7. Predict: Use model.predict(...) or call the model directly for inference.
8. Save: Store the complete model using model.save(...) or export it in the recommended deployment format.
9. Reload and verify: Load the saved model and confirm that its outputs remain consistent.

Reproducible data splits, fixed seeds where appropriate, and preserved preprocessing logic are also important.

Deep learning supports many applications, with architecture choice depending on data structure and task requirements.

• Computer vision: CNNs and vision transformers perform classification, object detection, segmentation, and medical-image analysis.
• Natural language processing: Transformers support translation, summarization, question answering, search, and text generation.
• Speech and audio: CNNs, recurrent models, and transformers enable speech recognition, speaker identification, and audio synthesis.
• Recommendation systems: Embedding models and deep ranking networks learn user-item relationships.
• Time-series analysis: Recurrent networks, temporal CNNs, and transformers support forecasting and anomaly detection.
• Generative systems: Autoencoders, GANs, diffusion models, and transformers generate images, audio, and text.
• Autonomous systems: Vision, sensor-fusion, and reinforcement-learning models assist perception and decision-making.

Deployment must also consider bias, privacy, explainability, reliability, energy use, and the consequences of incorrect predictions.