Unit 3: Learning, Adaptation and Fine-Tuning for LLMs and Agents - Subjective Questions

Supervised fine-tuning (SFT) is the process of adapting a pretrained language model using a labeled dataset containing input-output pairs. The model learns to generate the expected output for each input by minimizing a supervised loss.

For an input sequence and target response , the objective is commonly expressed as:

The main objectives of SFT are:

• Task adaptation: Specialize a general model for tasks such as summarization, classification, or question answering.
• Behavior adaptation: Teach the model to follow desired response patterns.
• Domain adaptation: Improve performance in domains such as medicine, law, or finance.
• Output consistency: Encourage responses that match labeled examples in structure, style, and content.

SFT updates the model parameters using gradients computed from the difference between generated and target outputs.

Instruction tuning is a form of supervised fine-tuning in which an LLM is trained on datasets containing natural-language instructions and corresponding desired responses. A training example may include an instruction, optional context, and an output.

It improves generalization by:

• Exposing the model to many task descriptions rather than only one fixed task.
• Teaching the model to infer the intended operation from natural-language instructions.
• Improving zero-shot performance on instructions not seen during training.
• Encouraging consistent formatting, relevance, and compliance.
• Helping the model transfer knowledge across related tasks.

For example, training on instructions for summarization, translation, classification, and reasoning teaches the model a general pattern: interpret the request and produce an appropriate response. Instruction tuning does not necessarily add new factual knowledge; it primarily improves how effectively the model uses its existing knowledge.

A typical supervised fine-tuning workflow contains the following stages:

1. Define the objective: Identify the task, target behavior, domain, and evaluation criteria.
2. Collect data: Obtain representative instruction-response or input-label examples.
3. Clean and validate data: Remove duplicates, incorrect labels, unsafe content, and low-quality examples.
4. Format examples: Convert records into a consistent prompt-response template.
5. Tokenize data: Transform text into token IDs and apply truncation or padding.
6. Split the dataset: Create training, validation, and test sets while avoiding data leakage.
7. Initialize the model: Load a suitable pretrained checkpoint.
8. Train the model: Minimize cross-entropy loss using an optimizer, learning-rate schedule, and regularization.
9. Monitor validation metrics: Detect overfitting using loss and task-specific measures.
10. Evaluate: Measure correctness, robustness, safety, and behavior on held-out examples.
11. Deploy and monitor: Track real-world failures and collect feedback for later improvement.

Careful dataset quality control is especially important because the model learns both the useful behavior and the errors present in its demonstrations.

The three stages differ in their data, objectives, and expected outcomes:

• Pretraining:

  • Uses very large, mostly unlabeled text corpora.
  • Commonly trains the model to predict the next token.
  • Develops broad language ability, factual patterns, and general representations.

• Supervised fine-tuning:

  • Uses labeled input-output examples.
  • Adapts the pretrained model to a task, domain, or desired behavior.
  • Usually uses less data and computation than pretraining.

• Instruction tuning:

  • Is a specialized form of SFT using natural-language instructions paired with responses.
  • Usually combines examples from multiple tasks.
  • Improves instruction following and transfer to unseen requests.

Thus, pretraining develops general capability, ordinary SFT specializes that capability, and instruction tuning teaches the model to apply its capability in response to diverse user instructions.

Catastrophic forgetting occurs when fine-tuning changes a model so strongly that it loses useful capabilities learned during pretraining. Overfitting occurs when the model memorizes the fine-tuning data and performs poorly on unseen examples.

Methods for reducing these problems include:

• Using a small learning rate to avoid large destructive parameter updates.
• Applying early stopping based on validation performance.
• Mixing general-purpose examples with domain-specific examples.
• Using parameter-efficient fine-tuning, such as LoRA, to preserve most pretrained weights.
• Applying regularization techniques such as weight decay and dropout.
• Increasing dataset diversity and removing duplicate examples.
• Limiting the number of epochs.
• Evaluating both the target task and important general capabilities.

A balanced training dataset is essential: a narrow or repetitive dataset can cause both memorization and loss of general behavior.

LoRA adapts a pretrained model while keeping its original weight matrix frozen. Instead of learning a full update , it represents the update as the product of two smaller matrices:

If , LoRA uses:

where the rank is much smaller than and .

LoRA is parameter-efficient because:

• Only and are trained.
• The pretrained weights remain frozen.
• Optimizer states and gradients are stored only for adapter parameters.
• Different adapters can be stored for different tasks.
• The update can often be merged into the base weights for deployment.

The method assumes that the useful task-specific weight update lies in a low-dimensional subspace.

In full fine-tuning, every element of is trainable. Therefore, the number of trainable parameters is:

LoRA represents the update as , where:

The two LoRA matrices contain:

and

parameters. Hence, the total number of trainable LoRA parameters is:

The ratio of LoRA parameters to fully trained parameters is:

For a square matrix with , this becomes:

For example, if and , full fine-tuning trains parameters, whereas LoRA trains only parameters. This is approximately of the full matrix.

The main LoRA design choices are:

• Rank : A larger rank gives the adapter more capacity to represent complex updates but increases memory, computation, and storage. A very small rank may underfit.
• Scaling: LoRA commonly scales the update as

where controls the contribution of the adapter. Scaling helps make training behavior more stable across different ranks.

• Target modules: LoRA may be applied to attention projections, feed-forward layers, or both. Adapting more modules increases expressiveness but also increases trainable parameters.
• Dropout: LoRA dropout can regularize adapter training and reduce overfitting.

An appropriate configuration depends on task complexity and available resources. Attention-only adaptation is efficient, while adapting attention and feed-forward projections may produce better results for difficult domain shifts.

Quantization represents model weights and, in some cases, activations using lower-precision numerical formats. For example, a model may be converted from 32-bit floating point to 16-bit, 8-bit, or 4-bit representations.

A simplified affine quantization rule is:

where is the original value, is the scale, is the zero point, and is the quantized integer.

Quantization supports deployment by:

• Reducing model storage requirements.
• Lowering memory bandwidth and RAM or VRAM usage.
• Allowing larger models to run on limited hardware.
• Potentially improving inference speed and energy efficiency.
• Reducing deployment cost.

The main drawback is quantization error, which may reduce accuracy or generation quality. Calibration, suitable granularity, and quantization-aware methods can reduce this degradation.

Post-training quantization (PTQ) converts an already trained model to lower precision without retraining all its parameters.

• It is fast and relatively inexpensive.
• It may use a calibration dataset to estimate ranges and scales.
• It is suitable when training resources are limited.
• Aggressive low-bit PTQ may cause noticeable quality loss.

Quantization-aware training (QAT) simulates quantization effects during training or fine-tuning.

• Fake-quantization operations expose the model to rounding and clipping errors.
• The model learns parameters that are more robust to low precision.
• It generally preserves accuracy better at very low bit widths.
• It requires additional training time, data, and computation.

Therefore, PTQ favors simplicity and low cost, while QAT favors better quality under demanding low-precision deployment constraints.

Quantized deployment involves several trade-offs:

• Memory versus quality: Lower bit widths save more memory but usually introduce more approximation error.
• Speed versus hardware support: Integer operations may be faster only when the target hardware and inference engine provide efficient kernels.
• Model size versus calibration effort: Advanced quantization can preserve quality but may require representative calibration data and careful tuning.
• Granularity versus overhead: Per-channel or group-wise quantization often improves accuracy, but it stores more scaling metadata and may complicate computation.
• Weights versus activations: Weight-only quantization is easier, while activation quantization can improve efficiency further but is sensitive to outliers.
• Static versus dynamic quantization: Static methods precompute ranges, whereas dynamic methods calculate some ranges at runtime and introduce overhead.

Evaluation should include task quality, latency, throughput, memory use, energy consumption, and compatibility with the target device rather than model size alone.

Q-learning is commonly formulated using a Markov decision process. Its main components are:

• State : A representation of the environment at a particular time.
• Action : A choice available to the agent in a state.
• Reward : A scalar feedback signal received after an action.
• Policy : A rule or probability distribution used to select actions.
• Action-value function : The expected discounted return from taking action in state and then following a policy.
• Discount factor : A value in controlling the importance of future rewards.

The discounted return from time is:

A small emphasizes immediate rewards, whereas a value close to gives greater importance to long-term consequences.

The optimal action-value function satisfies the Bellman optimality equation:

Because the true expectation is generally unknown, Q-learning updates its current estimate from an observed transition . The temporal-difference target is:

The temporal-difference error is:

The update rule is therefore:

Here:

• is the learning rate.
• is the immediate reward.
• discounts future rewards.
• estimates the best future value.
• The expression in brackets is the temporal-difference error.

Q-learning is off-policy because it learns the value of the greedy target policy even when actions are generated by an exploratory behavior policy.

The exploration-exploitation dilemma concerns whether an agent should:

• Exploit its current knowledge by selecting the action with the highest estimated value, or
• Explore other actions that may lead to better long-term outcomes.

Under an -greedy strategy:

• With probability , the agent chooses

• With probability , it chooses an action randomly.

A high initial encourages broad exploration. It is often gradually reduced so the agent increasingly exploits what it has learned. If decreases too quickly, the agent may converge to a poor policy; if it remains too high, behavior may stay unnecessarily random. Other exploration methods include softmax action selection, upper-confidence methods, and entropy-based exploration.

Reward shaping modifies or supplements the environment's reward signal to guide an agent toward useful behavior. For example, an agent may receive intermediate rewards for completing subgoals rather than receiving feedback only after final success.

Benefits include:

• Faster learning in environments with sparse rewards.
• Better credit assignment over long action sequences.
• Guidance toward safe or efficient behavior.
• Reduced exploration of clearly unproductive actions.

Risks include:

• Reward hacking: The agent exploits unintended shortcuts that maximize reward without achieving the real objective.
• Policy distortion: A poorly designed shaped reward changes the optimal behavior.
• Over-optimization: The agent focuses excessively on measurable proxies.
• Reduced generalization: Behavior may depend on artificial training signals unavailable during deployment.

Reward shaping should reflect the intended task while being tested for loopholes, side effects, and unintended strategies.

Potential-based reward shaping adds a shaping term derived from a potential function over states. The modified reward is:

For a trajectory beginning at , the discounted sum of shaping rewards is:

Expanding the terms produces a telescoping sum:

If is bounded and , the limit is zero. Thus, the shaped return differs from the original return only by , which is independent of the selected action at the initial state.

Consequently, action values are shifted consistently rather than reordered, preserving the optimal policy under standard assumptions. The potential function can still provide informative intermediate feedback and accelerate learning.

A sparse-reward problem occurs when meaningful feedback is received only rarely, such as after completing a long task. The credit-assignment problem is the difficulty of identifying which earlier actions caused a later success or failure.

These problems can be addressed through:

• Reward shaping: Supply informative intermediate rewards.
• Subgoals or curricula: Train on simpler stages before the complete task.
• Temporal-difference learning: Propagate later rewards back to earlier states.
• Eligibility traces or multi-step returns: Assign credit across multiple preceding actions.
• Demonstrations: Use expert trajectories to guide exploration.
• Intrinsic motivation: Reward novelty, curiosity, or progress.
• Hierarchical policies: Divide complex tasks into manageable skills.
• Human feedback: Evaluate partial trajectories or high-level behavior.

These methods must be designed carefully because excessive intermediate guidance can bias the agent toward unintended behavior.

A typical RLHF pipeline consists of the following stages:

1. Pretraining: Train a base language model on a large text corpus.
2. Supervised fine-tuning: Train the model on high-quality demonstrations of desired behavior.
3. Response generation: Produce multiple candidate responses or agent trajectories for selected prompts or tasks.
4. Human preference collection: Ask annotators to rank or compare candidates using criteria such as helpfulness, safety, and correctness.
5. Reward-model training: Train a model to assign higher scores to preferred outputs.
6. Policy optimization: Optimize the LLM or agent to maximize predicted reward, often while constraining deviation from a reference policy.
7. Evaluation: Test capability, safety, robustness, and resistance to reward hacking.
8. Iteration: Collect new feedback on failure cases and repeat the process.

A common objective includes a reward and a divergence penalty:

The penalty prevents the optimized policy from moving too far from a stable reference model.

A preference-based reward model learns a scalar score for response to input . Human annotators compare two or more candidate responses and indicate which one is preferred.

For a preferred response and a less-preferred response , the Bradley-Terry model defines the preference probability as:

where is the logistic sigmoid. The pairwise loss is:

Minimizing this loss teaches the reward model to assign higher scores to preferred outputs.

Important quality controls include:

• Clear annotation criteria.
• Multiple annotators for ambiguous examples.
• Measurement of inter-annotator agreement.
• Diverse prompts and candidate behaviors.
• Auditing for demographic, cultural, and positional biases.

The reward model approximates human preferences, so its errors can be exploited during policy optimization.

Supervised fine-tuning learns directly from demonstrated input-output or state-action examples. It is most useful when experts can provide clear examples of the desired behavior.

• Stable and relatively simple to train.
• Effective for teaching formats, tool-use patterns, and standard workflows.
• Limited by the cost and coverage of demonstrations.
• Does not directly optimize preferences among multiple acceptable behaviors.

Learning from human feedback uses comparisons, ratings, or corrections to define a reward or preference signal.

• Can represent qualitative goals that are difficult to demonstrate exactly.
• Helps select among several plausible actions or responses.
• Can optimize long-horizon agent behavior.
• Is vulnerable to noisy feedback, reward-model bias, and reward hacking.

A strong adaptation pipeline often combines both: SFT first teaches baseline behavior from demonstrations, and preference-based optimization then refines decisions according to human judgments. For agents, evaluation should cover task completion, tool correctness, safety, recovery from errors, and the long-term consequences of actions.