Unit 6: Artificial neural networks

Artificial neural networks (ANNs) are computational models inspired by the biological neuron, first formalised by McCulloch and Pitts (1943) and made trainable by the backpropagation algorithm (Rumelhart, Hinton & Williams, 1986). This unit treats the ANN as the substrate on which deep learning is built, then surveys network families, core algorithms, and their deployment in biology and healthcare.

I. Foundations: the artificial neuron and network learning

An ANN transforms inputs into outputs by composing many simple weighted units, and learning means adjusting weights to minimise a loss.

• Artificial neuron: computes a weighted sum plus bias passed through a nonlinearity:

z = Σ (wᵢ · xᵢ) + b
a = φ(z)

where xᵢ are inputs, wᵢ weights, b bias, φ the activation, a the output.

• Activation functions: introduce nonlinearity so the network can model non-linear boundaries.
  • Sigmoid: φ(z) = 1/(1+e⁻ᶻ), range (0,1), used for probabilities.
  • ReLU: φ(z) = max(0, z), cheap and resistant to vanishing gradients.
  • Softmax: normalises a vector into a probability distribution over classes.
• Layers: input, one or more hidden, and output; "deep" means many hidden layers.
• Loss function: measures error — mean squared error for regression, cross-entropy for classification.
• Learning rule: gradient descent updates each weight w ← w − η · ∂L/∂w, with η the learning rate.
• Deep vs shallow: shallow nets need hand-crafted features; deep nets learn a hierarchy of features automatically, the defining premise of deep learning.

II. Types of ANN

Architectural families and what each is built to model

Network topology is chosen to match the structure of the data — spatial, sequential, or generative.

A. Feedforward and fully connected networks

Signals flow one way from input to output with no cycles.

• Perceptron: single-layer linear classifier; solves only linearly separable problems (cannot learn XOR).
• Multilayer perceptron (MLP): stacked fully connected layers with nonlinear activations; a universal function approximator.
• Use case: tabular data such as gene-expression vectors mapped to a disease label.
• Limitation: ignores spatial or temporal ordering; parameter count explodes on images.

B. Convolutional neural networks (CNN)

Designed for grid-structured data by sharing weights across spatial positions.

• Convolution layer: slides a small kernel over the input to detect local patterns; weight sharing cuts parameters drastically.
• Pooling layer: downsamples (e.g. max-pool over 2×2) for translation invariance.
• Depth of features: early layers detect edges, deeper layers detect shapes and objects.
• Anchor: LeNet-5 (1998) for digits; ResNet (2015) introduced skip connections enabling 100+ layers.

C. Recurrent networks (RNN, LSTM, GRU)

Built for sequential data by carrying a hidden state across time steps.

1. Vanilla RNN: hₜ = φ(W·xₜ + U·hₜ₋₁ + b); suffers vanishing/exploding gradients over long sequences.
2. LSTM/GRU: add gates (input, forget, output) that regulate memory, retaining long-range dependencies.
   • Use case: protein sequences, ECG time series, clinical notes.

D. Autoencoders and generative networks

Learn compact representations or synthesise new data.

• Autoencoder: encoder compresses input to a latent code, decoder reconstructs it; loss is reconstruction error. Used for denoising and dimensionality reduction.
• Variational autoencoder (VAE): imposes a probabilistic latent space, enabling generation.
• Generative adversarial network (GAN): a generator and discriminator trained in opposition; the generator learns to produce realistic samples (e.g. synthetic medical images).

E. Transformers

Sequence models based on self-attention rather than recurrence.

• Self-attention: weighs the relevance of every token to every other, capturing long-range context in parallel.
• Anchor: the "Attention Is All You Need" architecture (2017); AlphaFold2 uses attention over residues.
• Advantage: parallelisable and scalable to very large datasets.

III. Introduction of deep learning algorithms

The training machinery shared across architectures

Deep learning algorithms are the procedures that fit network weights and control generalisation.

A. Forward propagation

Computes the network output for a given input.

• Process: apply layer transformations in sequence, a⁽ˡ⁾ = φ(W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾), until the output layer.
• Output: compared against the target to compute the loss L.

B. Backpropagation

Efficiently computes gradients of the loss with respect to every weight.

• Principle: applies the chain rule backward through the layers, reusing intermediate results.
• Core equation: error at a layer propagates as

δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾ᵀ δ⁽ˡ⁺¹⁾) ⊙ φ′(z⁽ˡ⁾)
∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ (a⁽ˡ⁻¹⁾)ᵀ

where δ is the local error, ⊙ elementwise product, φ′ the activation derivative.

• Cost: one forward and one backward pass per batch, linear in network size.

C. Optimisation algorithms

Decide how weights move given the gradients.

1. Stochastic gradient descent (SGD): updates on mini-batches; noisy but escapes shallow minima. Momentum adds a velocity term to smooth updates.
2. Adam: combines momentum with per-parameter adaptive learning rates using first and second moment estimates; the common default.
   • Learning-rate schedule: decay or warm-up stabilises convergence.

D. Regularisation and generalisation

Prevent the network from memorising training data.

• Dropout: randomly zeroes a fraction of units each step, forcing redundancy.
• L2 weight decay: penalises large weights, adding λ·Σw² to the loss.
• Batch normalisation: normalises layer inputs per mini-batch, speeding and stabilising training.
• Early stopping: halt when validation loss stops improving.

E. Applications and limitations of the algorithms

• Strengths: end-to-end feature learning, scale with data and compute, transferable via pretraining.
• Limitations: data-hungry, computationally costly, opaque ("black box"), and prone to overfitting on small biomedical cohorts.

IV. Case studies for the application of deep learning in biology and health care research

Concrete deployments and the architectures they rely on

Deep learning maps naturally onto biomedical data whose structure matches the network families above.

A. Protein structure prediction

Predicting 3D folds from amino-acid sequence.

• AlphaFold2 (DeepMind, 2020): attention-based network that won CASP14 with near-experimental accuracy.
• Input/output: residue sequence and evolutionary alignments in, inter-residue distances and angles out.
• Impact: structures for nearly all known proteins released, accelerating drug-target work.

B. Medical image diagnosis

Detecting disease from radiology and pathology images.

• Architecture: CNNs (e.g. Inception, ResNet) for classification; U-Net for segmentation.
• Anchor: diabetic-retinopathy detection from retinal fundus photographs reached specialist-level sensitivity; CNNs match dermatologists on skin-lesion classification.
• Segmentation: U-Net delineates tumour boundaries in MRI for radiotherapy planning.

C. Genomics and gene-expression analysis

Learning regulatory and functional signals from sequence and expression data.

• CNNs on DNA: treat the base sequence as a 1D signal to predict transcription-factor binding and splice sites (e.g. DeepBind, SpliceAI).
• MLP/autoencoder on expression: classify tumour subtypes or reduce single-cell RNA-seq dimensionality before clustering.
• Anchor: variant-effect prediction flags pathogenic mutations.

D. Drug discovery and generative design

Proposing and screening candidate molecules.

• Graph neural networks: represent molecules as graphs of atoms and bonds to predict binding affinity and toxicity.
• Generative models: VAEs and GANs generate novel molecular structures with desired properties.
• Impact: compresses early-stage screening from months to days.

E. Clinical time-series and electronic health records

Modelling patient trajectories for prediction.

• RNN/LSTM: ingest sequential vitals, labs, and diagnoses to forecast sepsis onset or readmission risk.
• Transformers on notes: clinical language models extract structured findings from free-text records.
• Signal example: CNN/RNN hybrids classify arrhythmias from raw ECG waveforms at cardiologist-level accuracy.

F. Significance and limitations in biomedicine

• Significance: surfaces patterns invisible to hand-crafted features, enabling earlier diagnosis and hypothesis generation.
• Data limitations: cohorts are small, imbalanced, and privacy-constrained; transfer learning and augmentation partly compensate.
• Interpretability: clinical adoption demands explanation methods (saliency maps, attention weights) so predictions can be trusted.
• Bias and validation: models trained on one population may fail on another, requiring external validation before deployment.