Unit 6: Artificial neural networks - Subjective Questions

An Artificial Neural Network (ANN) is a computational model inspired by the structure and functioning of biological neural networks in the human brain. It is composed of interconnected processing units called neurons (or nodes) that work together to learn patterns from data.

Basic Structure:

• Input Layer: Receives the raw input features. Each node represents one feature of the dataset.
• Hidden Layer(s): Perform intermediate computations. There can be one or many hidden layers; networks with many are called deep networks.
• Output Layer: Produces the final result (classification or regression output).

Fundamental Components:

• Weights (): Represent the strength of connections between neurons. They are adjusted during training.
• Bias (): An additional parameter that shifts the activation function.
• Activation Function: Introduces non-linearity, e.g. (sigmoid).
• Summation Function: Computes the weighted sum .

The neuron output is calculated as . Learning involves adjusting weights and biases to minimize prediction error.

There are several types of ANNs, each suited to specific tasks:

• Feedforward Neural Network (FNN): Information flows in one direction from input to output. No cycles. Application: Simple classification and regression tasks.
• Multilayer Perceptron (MLP): A feedforward network with one or more hidden layers using non-linear activation. Application: Pattern recognition.
• Convolutional Neural Network (CNN): Uses convolution layers to extract spatial features. Application: Image recognition, medical imaging.
• Recurrent Neural Network (RNN): Contains loops allowing information persistence, ideal for sequences. Application: Time-series analysis, DNA sequence modeling.
• Long Short-Term Memory (LSTM): A special RNN handling long-term dependencies. Application: Protein sequence prediction.
• Radial Basis Function Network (RBFN): Uses radial basis activation functions. Application: Function approximation.
• Autoencoders: Unsupervised networks that learn compressed representations. Application: Dimensionality reduction, anomaly detection.
• Generative Adversarial Networks (GAN): Two competing networks (generator and discriminator). Application: Synthetic medical image generation.

Each type is chosen based on the nature of the data and the problem to be solved.

The Perceptron is the simplest form of an artificial neuron, introduced by Frank Rosenblatt. It performs binary classification.

Working Steps:

1. Inputs: Receives multiple inputs .
2. Weighted Sum: Each input is multiplied by a corresponding weight and summed:
   
3. Activation: The sum passes through an activation function (e.g., step function):
   

Learning Rule: The weights are updated using:

where is the learning rate.

Limitation: A single perceptron can only solve linearly separable problems (e.g., it cannot solve the XOR problem). This limitation led to the development of multilayer networks.

Explanation:

• In an FNN, each input is processed independently, making it unsuitable for problems where order matters.
• In an RNN, the output of a neuron at time depends on both the current input and the previous hidden state: . This allows RNNs to model temporal dependencies, which is crucial for biological sequence analysis.

Deep Learning is a subset of Machine Learning that uses artificial neural networks with multiple hidden layers (deep architectures) to automatically learn hierarchical feature representations from large amounts of data.

Differences from Traditional Machine Learning:

• Feature Engineering:
  • Traditional ML: Requires manual feature extraction by domain experts.
  • Deep Learning: Automatically learns features from raw data.
• Data Requirements:
  • Traditional ML: Works well with smaller datasets.
  • Deep Learning: Requires large datasets to perform well.
• Hardware:
  • Traditional ML: Runs on standard CPUs.
  • Deep Learning: Often needs GPUs/TPUs for training.
• Performance:
  • Traditional ML: Performance plateaus with more data.
  • Deep Learning: Performance keeps improving with more data.
• Interpretability:
  • Traditional ML: More interpretable (e.g., decision trees).
  • Deep Learning: Often a "black box".

Deep learning has revolutionized fields such as computer vision, natural language processing, and computational biology.

Backpropagation is a supervised learning algorithm used to train multilayer neural networks by minimizing the error between predicted and actual outputs. It works by propagating the error backward through the network to update weights.

Key Steps:

1. Forward Pass: Compute the output by propagating inputs forward:
   

2. Compute Loss: Measure the error using a loss function, e.g., Mean Squared Error:
   

3. Backward Pass: Calculate gradients of the loss with respect to each weight using the chain rule:
   

4. Weight Update: Adjust weights using gradient descent:
   
   where is the learning rate.

5. Repeat: Iterate over many epochs until the error converges to a minimum.

Backpropagation efficiently computes gradients and is the foundation of training deep neural networks.

A Convolutional Neural Network (CNN) is a deep learning architecture specifically designed to process grid-like data such as images.

Architecture / Layers:

• Convolutional Layer: Applies filters (kernels) that slide over the image to detect features like edges and textures. The operation is:
  
• Activation (ReLU): Introduces non-linearity: .
• Pooling Layer: Reduces spatial dimensions (e.g., max pooling) to lower computation and control overfitting.
• Fully Connected Layer: Combines features to make the final classification.
• Output Layer: Uses softmax for multi-class prediction.

Application in Medical Image Analysis:

• Tumor Detection: CNNs classify MRI/CT scans to detect brain tumors or lung nodules.
• Diabetic Retinopathy: Analyzing retinal fundus images to detect disease severity.
• Histopathology: Detecting cancerous cells in tissue slides.
• Skin Cancer Detection: Classifying dermatological images as benign or malignant.

CNNs excel here because they automatically extract relevant spatial features, often achieving accuracy comparable to expert radiologists.

Activation functions introduce non-linearity into a neural network, enabling it to learn complex patterns. Without them, the network would behave like a simple linear model.

Common Activation Functions:

• Sigmoid: Maps values to .
  
  Drawback: Vanishing gradient problem.

• Tanh (Hyperbolic Tangent): Maps values to .
  

• ReLU (Rectified Linear Unit): Most widely used in deep networks.
  
  Advantage: Efficient and reduces vanishing gradient.

• Leaky ReLU: Allows small gradient for negative values.
  

• Softmax: Converts outputs into probabilities for multi-class classification.
  

The choice of activation function significantly affects training speed and model performance.

Suitability in Biological Research:

• CNN is suitable for:

  • Medical imaging (X-ray, MRI, CT scan analysis)
  • Histopathological image classification
  • Protein structure prediction from 2D contact maps
  • Cell/microscopy image segmentation

• RNN is suitable for:

  • DNA/RNA sequence analysis
  • Protein sequence modeling
  • Gene expression time-series analysis
  • Predicting binding sites in genomic sequences

Conclusion: CNNs handle structural/spatial biology data while RNNs (and LSTMs) handle sequential biological data. Hybrid CNN-RNN models are increasingly used for complex genomic tasks.

Deep learning has driven several breakthroughs in healthcare. Notable case studies include:

• Diabetic Retinopathy Detection (Google Health): A deep CNN was trained on retinal fundus images to detect diabetic retinopathy with accuracy comparable to ophthalmologists, enabling early screening.

• Skin Cancer Classification (Stanford): A CNN trained on ~130,000 skin images classified skin lesions at dermatologist-level accuracy for melanoma detection.

• AlphaFold (DeepMind): Used deep learning to predict 3D protein structures from amino acid sequences with remarkable accuracy, solving a decades-old problem in biology.

• Cancer Detection in Pathology: Deep learning models detect metastatic breast cancer in lymph node images (CAMELYON challenge), reducing pathologist workload.

• COVID-19 Diagnosis: CNNs analyze chest X-rays and CT scans to identify COVID-19 infection patterns.

• Drug Discovery: Deep neural networks predict molecular properties and drug-target interactions, accelerating pharmaceutical research.

Impact: These applications improve diagnostic accuracy, reduce costs, enable early detection, and accelerate biomedical discovery.

Gradient Descent is an optimization algorithm used to minimize the loss function by iteratively updating the model parameters in the direction of the steepest descent (negative gradient).

Update Rule:

where are parameters, is the learning rate, and is the loss function.

Variants:

• Batch Gradient Descent: Uses the entire dataset to compute the gradient each step.

  • Pros: Stable convergence. Cons: Slow for large datasets.

• Stochastic Gradient Descent (SGD): Updates parameters using one sample at a time.

  • Pros: Fast, can escape local minima. Cons: Noisy updates.

• Mini-batch Gradient Descent: Uses a small batch of samples. Balances speed and stability. Most commonly used.

Advanced Optimizers:

• Momentum: Accelerates convergence by adding a fraction of the previous update.
• RMSProp: Adapts learning rate per parameter.
• Adam: Combines Momentum and RMSProp; widely used in deep learning.

Choosing the right optimizer and learning rate is critical for effective training.

In deep networks, gradients are propagated backward through many layers using the chain rule. This can cause two problems:

Vanishing Gradient Problem:

• Occurs when gradients become extremely small as they propagate backward.
• Early layers learn very slowly or stop learning.
• Common with sigmoid/tanh activations because their derivatives are small.

Exploding Gradient Problem:

• Occurs when gradients grow exponentially large.
• Leads to unstable training and numerical overflow (NaN values).

Solutions:

• ReLU Activation: maintains gradients for positive inputs, reducing vanishing gradients.
• Weight Initialization: Techniques like Xavier and He initialization keep gradients balanced.
• Batch Normalization: Normalizes layer inputs to stabilize training.
• Gradient Clipping: Caps gradients to a maximum value to prevent exploding gradients.
• Residual Connections (ResNet): Skip connections allow gradients to flow directly, mitigating vanishing gradients.
• LSTM/GRU: Special gated architectures address vanishing gradients in RNNs.

These techniques enable stable training of very deep neural networks.

An Autoencoder is an unsupervised neural network that learns to compress (encode) input data into a lower-dimensional representation and then reconstruct (decode) it back.

Structure:

• Encoder: Compresses input into a latent representation : .
• Bottleneck (Latent Space): The compressed representation capturing essential features.
• Decoder: Reconstructs the input from : .

Objective: Minimize reconstruction error:

Types: Denoising autoencoders, sparse autoencoders, variational autoencoders (VAEs).

Applications in Bioinformatics:

• Dimensionality Reduction: Compressing high-dimensional gene expression data.
• Feature Extraction: Learning meaningful representations from genomic data.
• Anomaly Detection: Identifying abnormal cells or rare diseases.
• Drug Discovery: Generating molecular representations (with VAEs).
• Denoising: Cleaning noisy single-cell RNA sequencing data.

Autoencoders help manage the curse of dimensionality common in biological datasets.

LSTM (Long Short-Term Memory) is a special type of RNN designed to learn long-term dependencies and overcome the vanishing gradient problem of standard RNNs.

Core Components (Gates):

• Forget Gate: Decides what information to discard from the cell state.
  
• Input Gate: Decides what new information to store.
  
• Cell State Update: Updates the memory:
  
• Output Gate: Decides the output based on the cell state:
  
  

Role in Biological Sequences:

• DNA/Protein Sequences: LSTMs capture long-range dependencies between distant nucleotides or amino acids.
• Gene Expression Time Series: Model temporal dynamics of gene activity.
• Splice Site Prediction: Remember context across long sequences.

The gating mechanism allows LSTMs to selectively remember or forget information, making them ideal for sequential biological data where context spans long distances.

Overfitting: Occurs when a model learns the training data too well, including noise, and fails to generalize to new/unseen data. Characterized by low training error but high test error.

Underfitting: Occurs when a model is too simple to capture the underlying patterns in the data. Characterized by high training error and high test error.

Techniques to Prevent Overfitting:

• Regularization: Add penalty terms to the loss:
  • L1 (Lasso):
  • L2 (Ridge):
• Dropout: Randomly deactivate neurons during training to prevent co-adaptation.
• Early Stopping: Stop training when validation error starts increasing.
• Data Augmentation: Artificially increase dataset size (e.g., rotating/flipping images).
• Cross-Validation: Use k-fold validation to assess generalization.
• Reduce Model Complexity: Use fewer layers/neurons.
• Batch Normalization: Adds regularization effect.
• Increasing Training Data: More data helps the model generalize.

Balancing model complexity is key to achieving good generalization (the bias-variance tradeoff).

Deep learning has transformed genomics by automatically extracting patterns from massive sequence datasets.

Applications:

• Gene Prediction: CNNs and RNNs identify genes and regulatory regions within DNA sequences.

• Variant Calling: Google's DeepVariant uses CNNs to identify genetic variants (SNPs, indels) from sequencing data with high accuracy.

• Splice Site Prediction: Deep learning models predict exon-intron boundaries.

• Transcription Factor Binding: Tools like DeepBind predict where proteins bind to DNA/RNA sequences.

• Enhancer/Promoter Identification: Classifying regulatory elements in the genome.

• Protein Structure Prediction: AlphaFold predicts 3D protein structures from amino acid sequences.

• Gene Expression Prediction: Predicting expression levels from sequence data.

Why Deep Learning Works:

• Sequences are treated as strings that can be one-hot encoded.
• CNNs detect local motifs; RNNs/LSTMs capture long-range dependencies.

These approaches accelerate discovery in personalized medicine, disease diagnosis, and drug development.

Explanation:

• Supervised Learning: The network learns from examples with known outputs, minimizing error between predictions and true labels. Example: Disease classification from labeled medical images.

• Unsupervised Learning: The network finds structure in data without labels. Example: Clustering patients by gene expression.

• Reinforcement Learning: An agent learns optimal actions through trial and error to maximize reward. Example: Optimizing treatment strategies.

In biology, all three paradigms are used depending on data availability and problem type.

A Generative Adversarial Network (GAN) consists of two neural networks trained together in competition:

Components:

• Generator (G): Creates synthetic (fake) data from random noise, trying to fool the discriminator.
• Discriminator (D): Tries to distinguish real data from fake data generated by G.

Working (Minimax Game):

The two networks improve iteratively: the generator produces increasingly realistic data while the discriminator gets better at detection, until equilibrium.

Applications in Healthcare and Biology:

• Synthetic Medical Image Generation: Creating realistic MRI/CT images to augment limited datasets.
• Data Augmentation: Addressing class imbalance in rare diseases.
• Drug Discovery: Generating novel molecular structures with desired properties.
• Super-Resolution: Enhancing low-resolution medical scans.
• Privacy Preservation: Generating synthetic patient data that protects real identities.
• Denoising: Removing noise from biological images.

GANs help overcome data scarcity, a major challenge in medical and biological research.

A loss function (or cost function) quantifies the difference between the predicted output and the actual target. The goal of training is to minimize this loss.

Common Loss Functions:

• Mean Squared Error (MSE): Used for regression.
  

• Mean Absolute Error (MAE): Robust to outliers.
  

• Binary Cross-Entropy: Used for binary classification.
  

• Categorical Cross-Entropy: Used for multi-class classification.
  

• Hinge Loss: Used in SVM-like classifiers.
  

Importance:

• Guides the optimization process via gradients.
• The choice of loss depends on the task (regression vs classification).
• A well-chosen loss function leads to faster and more accurate convergence.

While deep learning offers powerful capabilities, its application in biology and healthcare faces several challenges:

Challenges and Limitations:

• Data Scarcity: Labeled medical data is limited and expensive to obtain, as annotation requires expert clinicians.

• Data Imbalance: Rare diseases have few positive samples, biasing models toward majority classes.

• Interpretability (Black Box): Deep models lack transparency, making it hard for doctors to trust or explain predictions—critical in clinical decisions.

• Data Privacy: Patient data is sensitive and subject to strict regulations (e.g., HIPAA, GDPR).

• Data Heterogeneity: Biological data comes from diverse sources with varying quality and formats.

• Overfitting: With small datasets, models may fail to generalize.

• Computational Cost: Training large models requires significant GPU resources.

• Validation & Regulatory Approval: Clinical deployment requires rigorous validation and regulatory clearance.

• Bias and Fairness: Models may inherit biases from training data, leading to health disparities.

Mitigation Strategies: Transfer learning, data augmentation, explainable AI (XAI), federated learning for privacy, and collaboration between AI experts and clinicians.

Addressing these challenges is essential for safe and effective deployment of deep learning in healthcare.