Back to Subject

Unit 1: Building Models with TensorFlow

INT422 — Deep Learning 10 min read

I. Orientation — The TensorFlow Model-Building Framework

TensorFlow is an open-source numerical computing and machine-learning framework originally developed by the Google Brain team and released in 2015. It represents data as tensors, performs mathematical operations on them, computes gradients automatically, and uses optimization algorithms to train models.

  • Core abstraction: A tensor is a multidimensional array whose elements usually share one data type, such as tf.float32.
  • Model-building principle: A model maps input tensors (X) to predicted outputs (\hat{Y}) through parameterized operations.
  • Learning principle: Training adjusts model parameters to minimize a loss function:
    [
    \theta^*=\arg\min_{\theta} L\big(Y,f(X;\theta)\big)
    ]
    where (\theta) denotes trainable parameters, (f) the model, (Y) targets, and (L) the loss.
  • Automatic differentiation: tf.GradientTape records operations and calculates derivatives needed for backpropagation.
  • Execution conventions:
    • Eager execution: Operations run immediately and return concrete values.
    • Graph execution: tf.function converts Python computations into optimized TensorFlow graphs.
  • High-level interface: tf.keras supplies layers, models, losses, metrics, optimizers, and training loops.
  • Hardware support: The same model can execute on CPUs, GPUs, or specialized accelerators such as TPUs.

II. TensorFlow Platform — Environment and Core Interface

TensorFlow provides interconnected APIs for numerical operations, data processing, automatic differentiation, model construction, training, deployment, and monitoring.

A. Introduction to TensorFlow

TensorFlow is designed to construct and execute differentiable computations efficiently across different hardware and deployment environments.

  • Tensor operations: Functions such as tf.add, tf.matmul, and tf.reduce_mean process tensors.
  • Keras integration: A neural network can be expressed as a sequence of layers:
    PYTHON
      import tensorflow as tf

  model = tf.keras.Sequential([
      tf.keras.layers.Dense(16, activation="relu"),
      tf.keras.layers.Dense(1)
  ])

    The first layer learns 16 nonlinear features; the final layer produces one output.
  • Automatic gradients: TensorFlow calculates derivatives without requiring symbolic differentiation by the programmer.
  • Data pipelines: tf.data.Dataset supports batching, shuffling, mapping, caching, and prefetching.
  • Model lifecycle: TensorFlow supports:
    • defining the model;
    • choosing a loss and optimizer;
    • fitting parameters;
    • evaluating performance;
    • saving and deploying the trained model.
  • Ecosystem: TensorFlow Lite targets mobile devices, while TensorFlow Serving supports production model serving.

B. Installation of TensorFlow

TensorFlow is normally installed inside an isolated Python environment to prevent dependency conflicts.

  • Environment creation: A virtual environment separates project packages:
    BASH
      python -m venv tf_env
  • Activation:
    BASH
      # Windows
  tf_env\Scripts\activate

  # macOS or Linux
  source tf_env/bin/activate
  • Package installation:
    BASH
      python -m pip install --upgrade pip
  pip install tensorflow
  • Verification:
    PYTHON
      import tensorflow as tf

  print(tf.__version__)
  print(tf.config.list_physical_devices())

    The first statement displays the installed version; the second lists available devices.
  • GPU use: TensorFlow uses a supported GPU when compatible drivers and required platform components are available.
  • Notebook environment: Jupyter can be added with pip install jupyter, although hosted notebook services may provide TensorFlow preinstalled.

III. Tensor Representation — Shape, Rank, and Transformation

TensorFlow represents scalars, vectors, matrices, images, sequences, and batches through one generalized tensor abstraction.

A. TensorFlow ranks and tensors

A tensor has a data type, shape, and rank; rank is the number of axes, not the number of stored elements.

  • Rank 0—scalar: A single value has shape ():
    PYTHON
      scalar = tf.constant(7)
  • Rank 1—vector: A one-dimensional collection has shape (n,):
    PYTHON
      vector = tf.constant([2.0, 4.0, 6.0])

    Its shape is (3,) and rank is 1.
  • Rank 2—matrix: Rows and columns produce shape (m, n):
    PYTHON
      matrix = tf.constant([[1, 2], [3, 4]])

    Its shape is (2, 2) and rank is 2.
  • Higher ranks: A color-image batch may have shape (32, 224, 224, 3), representing batch size, height, width, and channels.
  • Inspection:
    PYTHON
      print(matrix.shape)
  print(tf.rank(matrix))
  print(matrix.dtype)
  • Data types: Common types include tf.float32, tf.float64, tf.int32, and tf.bool; conversion uses tf.cast.
  • Immutability: Ordinary tensors are immutable values; changing a value creates a new tensor rather than modifying the original tensor.

B. Transforming tensors as multidimensional data arrays

Tensor transformations reorganize dimensions or values so that data matches the input and output requirements of model operations.

  • Reshaping: tf.reshape changes shape without changing element order:
    PYTHON
      x = tf.constant([1, 2, 3, 4, 5, 6])
  y = tf.reshape(x, (2, 3))

    The six elements become a (2 \times 3) matrix.
  • Transposition: tf.transpose permutes axes; a matrix of shape (2, 3) becomes (3, 2).
  • Dimension control:
    • tf.expand_dims(x, axis=0) inserts an axis.
    • tf.squeeze(x) removes axes of size 1.
  • Concatenation and stacking:
    • tf.concat([a, b], axis=0) joins existing axes.
    • tf.stack([a, b], axis=0) creates a new axis.
  • Slicing: Tensor syntax such as x[:, 0] selects all rows and the first column.
  • Broadcasting: Compatible smaller shapes are extended automatically; adding shape (3,) to shape (2, 3) applies the vector to each row.
  • Reduction: tf.reduce_sum, tf.reduce_mean, and tf.reduce_max aggregate values along selected axes.

IV. Computational Execution — Organizing Tensor Operations

TensorFlow computations can run interactively or be represented as reusable graphs that expose dependencies among operations.

A. TensorFlow's computation graphs

A computation graph represents operations as nodes and tensors flowing between operations as edges.

  1. Eager execution:

    • Behavior: Statements execute in Python order and results are immediately available.
    • Advantage: It simplifies debugging because tensor values can be inspected directly.
    • Example:
      PYTHON
           x = tf.constant(3.0)
     y = x ** 2 + 2 * x
     print(y.numpy())  # 15.0

  2. Graph execution:

    • Behavior: Decorating a function with @tf.function traces compatible TensorFlow operations into a graph.
    • Advantage: Graphs permit optimization, serialization, portability, and efficient execution.
    • Example:
      PYTHON
           @tf.function
     def predict(x, w, b):
         return tf.matmul(x, w) + b
    • Dependencies: In (y=Wx+b), matrix multiplication must produce (Wx) before addition can generate (y).
    • Tracing consideration: Python-side effects and changing input structures may cause unexpected behavior or repeated graph tracing.
    • Control flow: TensorFlow can convert compatible conditional and loop constructs into graph operations.

V. Model State — Storing Learnable Parameters

Neural networks require mutable state for weights, biases, counters, and other values that change during execution.

A. Variables in TensorFlow

A tf.Variable stores mutable tensor data and can be updated while preserving its identity.

  • Creation:
    PYTHON
      weight = tf.Variable([[0.2], [-0.4]], dtype=tf.float32)
  bias = tf.Variable([0.0], dtype=tf.float32)
  • Mutation:
    • assign(value) replaces the stored value.
    • assign_add(value) adds to it.
    • assign_sub(value) subtracts from it.
  • Trainability: Variables use trainable=True by default, allowing optimizers to update them.
  • Layer parameters: A Dense layer typically maintains a kernel (W) and bias (b):
    [
    Z=XW+b
    ]
    where (X) is the input matrix and (Z) is the pre-activation output.
  • Initialization: Random, zero, or variance-scaled initializers establish starting values; initialization affects convergence and gradient stability.
  • Gradient tracking: tf.GradientTape automatically watches trainable variables used inside its context.
  • Persistence: Variables are included when Keras model weights or TensorFlow checkpoints are saved.

VI. Model Training — Gradient-Based Parameter Updates

Training repeatedly computes predictions, measures error, differentiates that error, and updates variables.

A. TensorFlow optimizers

An optimizer applies calculated gradients to variables in order to reduce the selected loss function.

  • Gradient descent rule:
    [
    \theta_{t+1}=\thetat-\eta\nabla{\theta}L(\thetat)
    ]
    where (t) is the update step, (\eta) the learning rate, and (\nabla    {\theta}L) the loss gradient.
  • Stochastic Gradient Descent: tf.keras.optimizers.SGD performs direct gradient updates and may include momentum.
  • Adam: tf.keras.optimizers.Adam maintains moving estimates of first and second gradient moments, often providing fast practical convergence.
  • Learning rate: A value that is too large may cause divergence; one that is too small may make training slow.
  • Custom training step:
    PYTHON
      optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

  with tf.GradientTape() as tape:
      predictions = model(x, training=True)
      loss = tf.reduce_mean((y - predictions) ** 2)

  gradients = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))
  • Compiled training: model.compile(optimizer="adam", loss="mse") connects an optimizer and loss to Keras training methods.
  • Optimization limits: Optimizers cannot compensate for invalid data, unsuitable model architecture, or a loss function misaligned with the task.

VII. Training Visualization — Monitoring Experiments

Effective model development requires inspecting losses, metrics, graph structure, parameter distributions, and computational performance.

A. Visualization with TensorBoard

TensorBoard is TensorFlow’s visualization toolkit for tracking model behavior across training runs.

  • Scalar dashboards: Display values such as training loss, validation loss, accuracy, and learning rate over epochs.
  • Graph dashboard: Shows the structure and connectivity of TensorFlow operations.
  • Histogram dashboard: Displays changes in weights, biases, activations, or gradients over time.
  • Profiler: Identifies execution bottlenecks, device utilization, and expensive operations.
  • Keras callback:
    PYTHON
      callback = tf.keras.callbacks.TensorBoard(
      log_dir="logs/run_1",
      histogram_freq=1
  )

  model.fit(x_train, y_train, epochs=10, callbacks=[callback])
  • Launching TensorBoard:
    BASH
      tensorboard --logdir logs
  • Comparative use: Separate log directories allow runs with different learning rates, architectures, or batch sizes to be compared.
  • Interpretation: Falling training loss with rising validation loss is concrete evidence of overfitting rather than continued generalization.

VIII. Deep Learning — Representation Learning and Use Cases

Deep learning uses multilayer neural networks to learn hierarchical representations directly from data, reducing dependence on manually designed features.

A. Introduction to Deep Learning

Deep learning models organize many parameterized layers so that simple lower-level patterns combine into increasingly abstract features.

  • Artificial neuron:
    [
    a=\phi\left(\sum_{i=1}^{n}w_ix_i+b\right)
    ]
    where (x_i) is an input, (w_i) its weight, (b) the bias, (\phi) an activation function, and (a) the output.
  • Hidden layers: Intermediate layers learn representations; image layers may progress from edges to textures, parts, and objects.
  • Nonlinearity: Activations such as ReLU, (\max(0,z)), allow networks to approximate nonlinear relationships.
  • Forward propagation: Inputs move through successive layers to produce predictions.
  • Backpropagation: The chain rule propagates loss derivatives backward through the network.
  • Training requirements: Effective systems usually depend on representative data, suitable architecture, computational resources, and regularization.
  • Major architectures: Convolutional networks process spatial data, recurrent or sequence models process ordered data, and transformers use attention to model relationships.

B. Applications of Deep Learning

Deep learning is most effective where large datasets contain complex patterns that are difficult to encode manually.

  • Computer vision: Convolutional and vision-transformer models support image classification, object detection, segmentation, and medical-image analysis.
  • Natural language processing: Transformer models perform translation, summarization, sentiment analysis, question answering, and text generation.
  • Speech and audio: Networks enable speech recognition, speaker identification, sound classification, and speech synthesis.
  • Recommendation systems: Learned user and item representations help rank products, films, music, or advertisements.
  • Healthcare: Models can assist with scan interpretation, risk estimation, drug discovery, and physiological-signal analysis.
  • Industrial systems: Sensor and image data support predictive maintenance, defect detection, and process monitoring.
  • Autonomous systems: Deep networks contribute to perception, localization, planning, and control in vehicles and robots.
  • Limitations: Applications must address biased data, weak interpretability, privacy risks, high computational cost, distribution shifts, and confidently incorrect predictions.
Next
Unit 2