Unit 3: Learning, Adaptation and Fine-Tuning for LLMs and Agents

I. Orientation — Adapting Pretrained Models

Large language models (LLMs) first learn broad statistical patterns through pretraining on large text corpora. Adaptation then changes model behavior for particular instructions, domains, deployment constraints, or agent objectives without necessarily retraining the entire model.

• Governing principle: Adaptation optimizes model parameters or decision policies against a task-specific objective while attempting to preserve useful pretrained capabilities.
• Core learning settings:
  • Supervised learning: Learns from input–target pairs, such as an instruction paired with an ideal response.
  • Reinforcement learning (RL): Learns actions from scalar rewards received through interaction with an environment.
  • Human-feedback learning: Converts human preferences, demonstrations, or corrections into training signals.
• LLM objective: An autoregressive model estimates the probability of token (y_t) from earlier tokens:

P(y₁, ..., y_T | x) = ∏ₜ₌₁ᵀ P(yₜ | x, y₁, ..., yₜ₋₁)

• (x): input context or instruction.
• (y_t): target token at position (t).
• (T): number of output tokens.
  • Agent objective: An agent chooses actions through a policy (\pi(a\mid s)), where (s) is the current state and (a) is an action, to maximize cumulative reward.
  • Practical constraints: Adaptation must balance model quality, computational cost, memory usage, latency, safety, and resistance to catastrophic forgetting.

II. Supervised Adaptation — Learning Desired Responses

A. Supervised fine-tuning and instruction tuning

Supervised fine-tuning (SFT) updates a pretrained model using labelled examples, while instruction tuning is SFT specifically designed to improve the model’s response to natural-language instructions.

• Training data: Each example normally contains an input (x) and a desired output (y).
  • For classification, (y) may be a label such as positive.
  • For generation, (y) may be a complete assistant response.
  • Instruction datasets commonly use fields such as instruction, input, and response.
• Optimization objective: Teacher forcing supplies the correct preceding target tokens and minimizes token-level negative log-likelihood:

L_SFT(θ) = -∑ₜ₌₁ᵀ log Pθ(yₜ | x, y₁, ..., yₜ₋₁)

• (\theta): trainable model parameters.
• (P_\theta): probability assigned by the model.
• (y_t): correct token at output position (t).
  • Supervised fine-tuning: Domain-specific examples adapt terminology, response structure, or task competence; medical-document summarization, for example, requires representative documents and approved summaries.
  • Instruction tuning: Training across diverse tasks teaches the general pattern “interpret instruction, observe constraints, produce an appropriate answer.” Prompt variety is important because repeated templates can cause brittle behavior.
  • Loss masking: In assistant training, prompt tokens may be excluded from the loss so that optimization focuses on response tokens rather than reproducing the user’s text.
  • Data quality: Deduplication, consistent formatting, correct labels, safety screening, and train–validation separation often matter more than simply increasing example count.
  • Risks and controls:
• Overfitting: Validation loss may rise while training loss falls; early stopping and data diversity help.
• Catastrophic forgetting: Narrow training can weaken general abilities; lower learning rates or mixing general data can reduce this effect.
• Exposure bias: Training sees correct prefixes, whereas inference sees the model’s own generated prefixes.

III. Efficient Adaptation — Updating a Small Parameter Set

A. Parameter-efficient techniques (LoRA)

Parameter-efficient fine-tuning adapts a large model by training a small set of added or selected parameters; Low-Rank Adaptation (LoRA) represents weight updates through low-rank matrices.

• Low-rank update: For a frozen weight matrix (W_0), LoRA learns an update (\Delta W):

W = W₀ + ΔW
ΔW = (α/r)BA

• (W0 \in \mathbb{R}^{d{out}\times d_{in}}): original frozen weight.
• (A \in \mathbb{R}^{r\times d{in}}) and (B \in \mathbb{R}^{d{out}\times r}): trainable matrices.
• (r): rank, usually much smaller than (d{in}) and (d{out}).
• (\alpha): scaling factor controlling update magnitude.
  • Parameter saving: Full tuning trains (d{out}d{in}) parameters for the matrix, whereas LoRA trains (r(d{in}+d{out})).
  • Placement: LoRA adapters are commonly attached to Transformer attention projections—such as query, key, value, or output matrices—and sometimes to feed-forward layers.
  • Initialization: One low-rank factor is typically initialized so that (\Delta W=0) initially, preserving the base model’s starting behavior.
  • Operational benefit: Multiple small adapters can share one base model, enabling separate domain or customer adaptations without storing multiple complete checkpoints.
  • Deployment: The update can remain as an adapter or be merged into (W_0) for inference, avoiding an additional adapter computation.
  • Limitations: A very small rank may underfit complex changes, while adapter quality still depends on data quality, target-module selection, learning rate, and compatibility with the base checkpoint.

IV. Model Compression — Reducing Numerical Precision

A. Quantization for deployment

Quantization represents model weights or activations with fewer bits to reduce memory consumption, bandwidth demand, and often inference latency.

• Basic mapping: Uniform affine quantization maps a real value (x) to an integer (q):

q = clamp(round(x/s) + z, q_min, q_max)
x̂ = s(q - z)

• (s): positive scale.
• (z): zero-point mapping real zero into the integer range.
• (q{min}, q{max}): limits of the target integer format.
• (\hat{x}): reconstructed approximation of (x).
  • Precision choices: FP16 and BF16 reduce memory relative to FP32; INT8 and 4-bit formats compress more aggressively but introduce larger approximation errors.
  • Quantized components:
• Weight-only quantization: Stores weights at low precision while activations use higher precision.
• Weight-and-activation quantization: Can accelerate supported integer kernels but requires activation calibration.
  • Granularity: Per-tensor quantization uses one scale for a tensor; per-channel or group-wise quantization uses multiple scales and usually preserves outlier-sensitive weights more accurately.
  • Post-training quantization: Quantizes an already-trained model using calibration data and requires no full retraining.
  • Quantization-aware training: Simulates rounding and clipping during training so parameters can adapt to quantization error.
  • Deployment trade-off: A model with (N) parameters requires approximately (4N) bytes in FP32 but (0.5N) bytes for ideal 4-bit weight storage, excluding scales, metadata, activations, and runtime buffers.
  • Limitations: Actual speed depends on hardware kernels; excessive compression can degrade perplexity, reasoning, rare-token prediction, or output stability.

V. Reinforcement Learning — Learning from Consequences

A. Reinforcement learning foundations (Q-learning basics)

Reinforcement learning models sequential decision-making, and Q-learning estimates the long-term value of taking an action in a state without requiring a known environment model.

• Markov decision process: An MDP is commonly represented by ((S,A,P,R,\gamma)).
  • (S): set of states.
  • (A): set of actions.
  • (P(s'\mid s,a)): transition probability.
  • (R(s,a,s')): immediate reward.
  • (\gamma\in[0,1)): discount factor.
• Return: The discounted reward from time (t) is:

Gₜ = rₜ₊₁ + γrₜ₊₂ + γ²rₜ₊₃ + ...

• Action value: (Q(s,a)) estimates expected return after action (a) in state (s), followed by future policy decisions.
• Q-learning update:

Q(s,a) ← Q(s,a) + η[r + γ maxₐ′ Q(s′,a′) - Q(s,a)]

• (\eta): learning rate.
• (r): observed immediate reward.
• (s'): next state.
• (a'): candidate next action.
• The bracketed expression is the temporal-difference error.
  • Worked update: If (Q(s,a)=2), (\eta=0.5), (r=1), (\gamma=0.9), and (\max_{a'}Q(s',a')=4), then the target is (1+0.9(4)=4.6), giving the new value (2+0.5(4.6-2)=3.3).
  • Exploration: An (\varepsilon)-greedy policy chooses a random action with probability (\varepsilon) and the currently highest-valued action otherwise.
  • LLM-agent relevance: States may include conversation and tool history, while actions may be messages or tool calls. Tabular Q-learning is usually impractical because these spaces are enormous, motivating function approximation and policy-based methods.

VI. Reward Design — Directing Agent Learning

A. Reward shaping

Reward shaping supplements or transforms environmental rewards so that useful intermediate behavior receives a stronger and less sparse learning signal.

• Sparse-reward problem: If an agent receives (+1) only after completing a long workflow, it receives little information about which intermediate actions—such as selecting the correct tool—were useful.
• Shaped reward: A task reward can be combined with auxiliary terms:

r′ = r_task + λ₁r_progress - λ₂c_tool - λ₃c_unsafe

• (r_{task}): reward for task completion.
• (r_{progress}): measurable movement toward the goal.
• (c_{tool}): cost of unnecessary tool use.
• (c_{unsafe}): penalty for unsafe behavior.
• (\lambda_1,\lambda_2,\lambda_3): weighting coefficients.
  • Potential-based shaping: Adding (F(s,a,s')=\gamma\Phi(s')-\Phi(s)), where (\Phi) is a state-potential function, preserves the optimal policy under standard MDP assumptions.
  • Agent signals: Useful measures can include verified task success, valid API arguments, citation correctness, latency, number of tool calls, or constraint compliance.
  • Reward hacking: An agent may maximize the measured proxy without achieving the intended goal—for example, producing many superficially valid steps to collect progress rewards.
  • Design controls: Use externally verifiable outcomes, cap repeatable bonuses, test adversarial trajectories, and keep safety constraints separate from easily traded-off convenience rewards.

VII. Preference-Based Alignment — Incorporating Human Judgement

A. Learning from human feedback for agent behavior

Learning from human feedback uses demonstrations, rankings, ratings, or corrections to make model and agent behavior better match human intentions when automatic objectives are incomplete.

• Demonstrations: Human-written trajectories can supervise response generation, planning, tool selection, argument construction, and stopping decisions through SFT.
• Preference collection: Evaluators compare candidate outputs or trajectories for the same prompt, producing pairs such as preferred (y_w) and rejected (y_l).
• Reward modelling: A reward model (r_\phi(x,y)) can be trained with a pairwise preference loss:

L_RM(φ) = -log σ(rφ(x,y_w) - rφ(x,y_l))

• (\phi): reward-model parameters.
• (\sigma): logistic sigmoid.
• (x): prompt or initial state.
• (y_w,y_l): preferred and rejected outputs.
  • Policy optimization: The agent is optimized to obtain higher learned reward, commonly with a penalty for moving too far from a reference policy:

Objective = E[rφ(x,y)] - β D_KL(πθ || π_ref)

• (\pi_\theta): adapted policy.
• (\pi_{ref}): reference policy, often the SFT model.
• (D_{KL}): Kullback–Leibler divergence.
• (\beta): strength of behavior-preservation regularization.
  • Agent-level feedback: Evaluating complete trajectories captures whether the agent chose appropriate tools, recovered from errors, avoided harmful actions, and stopped after verified completion—not merely whether its final prose sounded convincing.
  • Feedback quality: Clear rubrics, multiple evaluators, disagreement monitoring, and expert review improve consistency; biased or ambiguous judgements are inherited by the learned objective.
  • Failure modes: Reward-model exploitation, excessive agreeableness, verbosity, evaluator bias, and distribution shift can produce high predicted reward without genuine usefulness.
  • Safety measures: Combine preference learning with hard permission boundaries, sandboxed tools, outcome verification, adversarial evaluation, and human approval for consequential actions.