Unit 6: Evaluation, Safety and Deployment of LLM and Agentic Systems

I. Foundations of Reliable Agentic AI

An LLM-driven agent is a system in which a large language model selects actions, uses tools, observes results, maintains state, and iterates toward a goal. Reliable deployment therefore requires evaluating the entire decision loop—not merely the linguistic quality of the model’s final response.

• Defining properties:
  • Autonomy: The system can choose intermediate actions, such as calling a search API or querying a database, without a human specifying every step.
  • Tool use: Actions interact with external components such as code interpreters, retrieval systems, browsers, and transaction APIs.
  • Statefulness: Memory, conversation history, tool outputs, and environmental observations influence later decisions.
  • Non-determinism: Sampling, changing external data, and tool failures can produce different trajectories from the same initial request.
• Reliability objectives:
  • Capability: The system completes the intended task correctly.
  • Safety: It avoids harmful, prohibited, or unauthorized actions.
  • Robustness: It behaves acceptably under malformed inputs, failures, and adversarial attacks.
  • Accountability: Decisions can be traced, reviewed, and assigned to responsible stakeholders.
  • Operational fitness: Latency, throughput, availability, and cost satisfy production requirements.
• Core evaluation unit: An agent trajectory can be represented as
  (\tau=(s_0,a_0,o1,\ldots,a{T-1},o_T)), where (s_0) is the initial state, (at) is an action, (o{t+1}) is the resulting observation, and (T) is the number of steps.
• Lifecycle principle: Evaluation, safety, deployment, and monitoring form a continuous feedback loop; production incidents must generate new tests, policies, and training data.

II. Evaluation of Agent Performance — Measuring Outcomes and Behaviour

Evaluation determines whether an agent reaches valid outcomes through safe, efficient, and reproducible trajectories.

A. Metrics for evaluating LLM-driven agents

Agent metrics should measure final-task success, intermediate decisions, operational efficiency, and risk rather than relying on text similarity alone.

• Task success rate: The fraction of tasks satisfying an externally verifiable completion criterion.

Success Rate = N_success / N_total

Here, (N{\text{success}}) is the number of correctly completed tasks and (N{\text{total}}) is the total number evaluated.

• Answer-quality metrics:
  • Exact match or F1: Suitable for constrained answers, extraction, and classification.
  • Semantic or rubric scoring: Useful for open-ended outputs, preferably with calibrated human review.
  • Groundedness: Measures whether claims are supported by retrieved evidence.
• Trajectory metrics:
  • Step efficiency: Compares actions taken with the shortest acceptable trajectory.
  • Tool-call accuracy: Checks tool selection, argument validity, and interpretation of returned values.
  • Recovery rate: Measures successful completion after tool errors or invalid observations.
• Safety metrics:
  • Violation rate: Fraction of runs containing prohibited content or actions.
  • Attack success rate: Fraction of adversarial prompts that bypass controls.
  • Unauthorized-action rate: Frequency of operations outside the agent’s permissions.
• Operational metrics: End-to-end latency, tokens per task, tool cost, throughput, and failure rate reveal whether capability is economically deployable.
• Composite scoring: A weighted score may be written as:

S = wqQ + wsR + weE - wcC

(S) is total score; (Q) is quality, (R) is safety compliance, (E) is efficiency, (C) is cost, and each (w) is a policy-defined weight. Separate component values should also be reported because a single score can conceal critical safety failures.

B. Testing approaches

Testing must examine components, complete trajectories, adversarial conditions, and real production behaviour.

• Unit testing: Tests deterministic components such as prompt templates, policy functions, output parsers, tool schemas, and permission checks.
• Integration testing: Verifies interactions among the model, retrieval layer, memory store, tools, authentication, and external APIs.
• End-to-end testing: Runs representative tasks from user input to final outcome in a sandbox with explicit success criteria.
• Regression testing: Replays a fixed benchmark after model, prompt, tool, or policy changes; results should be compared by task category rather than only by aggregate score.
• Adversarial testing: Uses prompt injection, jailbreaks, poisoned documents, malformed tool responses, and excessive-context attacks to expose control failures.
• Simulation and fault injection: Introduces timeouts, rate limits, unavailable tools, stale data, and partial responses to assess recovery behaviour.
• Human evaluation: Domain experts apply a documented rubric to correctness, relevance, evidence, and harmfulness; inter-rater agreement helps assess rubric consistency.
• Statistical discipline: Test sets must include ordinary, edge, and high-risk cases; confidence intervals and repeated runs are important because agents are non-deterministic.

III. Safety and Governance — Controlling Risk and Responsibility

Safety engineering limits harmful inputs, outputs, and actions, while governance defines who sets those limits and answers for failures.

A. Safety guardrails and prompt filtering

Guardrails should provide layered controls before inference, during planning, at tool boundaries, and after generation.

• Input filtering: Classifiers and deterministic rules detect prohibited requests, sensitive data, malicious payloads, and known injection patterns.
• Prompt-injection defence: Retrieved text must be treated as untrusted data, separated from system instructions, and prevented from redefining permissions.
• Output filtering: Generated content can be checked for policy violations, personal data leakage, unsupported claims, and unsafe executable instructions.
• Tool guardrails:
  • Allowlisting: Expose only tools required for the assigned role.
  • Schema validation: Reject missing, malformed, or out-of-range arguments.
  • Least privilege: Use narrowly scoped credentials and read-only access where possible.
  • Confirmation gates: Require human approval for payments, deletion, publication, or other consequential actions.
• Constrained generation: JSON schemas, grammars, fixed action spaces, and parameter bounds reduce syntactic and behavioural freedom.
• Defence in depth: No classifier is perfect; independent controls should prevent one bypass from becoming a harmful action.
• Safe failure: When policy status, authorization, or tool output is uncertain, the agent should abstain, request clarification, or escalate rather than guess.

B. Ethical considerations and Responsible AI guidelines

Responsible AI translates values such as fairness, privacy, transparency, and human oversight into enforceable system requirements.

• Fairness: Performance and error rates should be evaluated across relevant populations; proxy variables can reproduce discrimination even when protected attributes are removed.
• Privacy: Data minimization, purpose limitation, retention schedules, access controls, and redaction reduce unnecessary exposure of personal information.
• Transparency: Users should know when they are interacting with AI, what major limitations apply, and when outputs are generated rather than human-authored.
• Human agency: People need meaningful routes to review, override, appeal, or correct consequential decisions.
• Accountability: Model owners, deployers, tool providers, and operators require documented responsibilities and incident-escalation paths.
• Proportionality: Stronger controls are necessary as potential harm increases; an entertainment assistant and a medical triage assistant should not share the same approval threshold.
• Governance artefacts: Model cards, system cards, risk registers, impact assessments, audit logs, and change approvals make responsible-AI commitments operational.
• Dual-use risk: Teams should assess both intended benefits and foreseeable misuse, especially when agents can execute code, access private data, or operate at scale.

IV. Interpretability and Accountability — Understanding Agent Decisions

Interpretability makes autonomous behaviour reviewable without assuming that generated explanations perfectly reveal internal model reasoning.

A. Explainability of autonomous decisions

Useful explanations connect goals, evidence, actions, and outcomes through auditable records and faithful summaries.

• Decision provenance: Record the user request, policy version, model version, retrieved document identifiers, tool calls, approvals, and final result.
• Trajectory explanation: Present concise action-level reasons such as “queried inventory because stock status was required,” rather than exposing unrestricted hidden reasoning.
• Evidence attribution: Claims should link to source passages or structured records; citation presence alone is insufficient unless the cited material supports the claim.
• Local explanation: Explains one decision—for example, which retrieved facts caused an application to be escalated.
• Global explanation: Describes system-wide behaviour, including available tools, permission boundaries, refusal policies, and known failure modes.
• Faithfulness problem: An LLM can generate a plausible post-hoc rationale unrelated to the actual computation; explanations should therefore be anchored in logged inputs and actions.
• Counterfactual analysis: Testing whether a changed fact alters the result can reveal decision sensitivity, such as whether removing an irrelevant demographic proxy changes a ranking.
• Privacy balance: Logs must support audits without retaining secrets, unnecessary personal data, or raw chain-of-thought content.

V. Production Infrastructure — Serving, Observing, and Operating Agents

Production infrastructure must deliver model inference and multi-step workflows predictably under variable demand, failures, and evolving risks.

A. Scalable serving frameworks

Scalable serving frameworks coordinate model execution, hardware resources, request scheduling, and agent workflows.

• Inference servers: Systems such as vLLM, NVIDIA Triton, and Hugging Face Text Generation Inference support batching, streaming, parallelism, and optimized memory use.
• Continuous batching: Requests are added and removed dynamically during generation, improving accelerator utilization despite differing sequence lengths.
• Caching: Prefix caching reuses repeated prompt computation, while semantic caching may reuse answers only when freshness and authorization permit.
• Horizontal scaling: Load balancers distribute requests across replicas; autoscaling can use queue depth, token throughput, latency, and accelerator utilization.
• Model optimization: Quantization reduces numerical precision and memory use; tensor or pipeline parallelism distributes large models across devices.
• Agent orchestration: Workflow engines should support durable state, retries, timeouts, idempotency, and cancellation across long-running tool calls.
• Service objectives: Tail latency such as p95 or p99 is more informative than averages because a small number of slow multi-step runs can harm user experience.
• Trade-off: Larger models may improve capability but increase latency and cost; routing simple tasks to smaller models can preserve quality while controlling resource use.

B. Monitoring systems

Monitoring must combine conventional service telemetry with model-quality, agent-trajectory, and safety signals.

• Operational telemetry: Track availability, request rate, error rate, queue depth, latency percentiles, token usage, accelerator utilization, and tool timeouts.
• Quality telemetry: Sample groundedness, task success, refusal correctness, malformed outputs, repeated loops, and human-escalation frequency.
• Safety telemetry: Monitor policy violations, injection attempts, sensitive-data exposure, unusual tool calls, and blocked high-impact actions.
• Distributed tracing: A trace identifier should connect model calls, retrieval operations, tool invocations, retries, and approvals for one task.
• Drift detection: Changes in user inputs, retrieval sources, tool behaviour, costs, or success rates can indicate that offline benchmarks no longer represent production.
• Alert design: Alerts need thresholds, severity levels, responsible owners, and runbooks; excessive low-value alerts create fatigue.
• Privacy-preserving observability: Redact secrets and personal data, restrict log access, encrypt retained records, and apply explicit deletion periods.
• Feedback loop: Confirmed failures should become regression cases, while monitoring dashboards should distinguish model, orchestration, tool, and policy failures.

C. Principles for deploying reliable LLM and agentic systems

Reliable deployment depends on bounded autonomy, staged release, reversible change, and continuous evidence of acceptable behaviour.

• Risk-based design: Classify use cases by impact and prohibit full autonomy where errors could cause unacceptable or irreversible harm.
• Least agency: Grant only the tools, data, permissions, action budget, and runtime necessary for the task.
• Staged rollout: Progress through offline evaluation, sandboxing, shadow mode, internal use, canary release, and controlled expansion.
• Reversibility: Maintain versioned prompts, models, policies, and tool schemas, with feature flags, rollback procedures, and kill switches.
• Failure containment: Apply rate limits, spending limits, maximum step counts, isolated execution environments, and transactional boundaries.
• Human oversight: Escalation should occur when confidence is low, policy is ambiguous, requested actions are consequential, or repeated recovery fails.
• Reproducibility: Record configuration versions, model identifiers, sampling parameters, and dependency changes so incidents can be reconstructed.
• Security posture: Authenticate users and services, authorize every tool action, rotate credentials, isolate tenants, and test supply-chain dependencies.
• Continuous assurance: Re-evaluate after changes to models, prompts, data, policies, tools, or user populations; passing a pre-release benchmark is not permanent certification.
• Incident readiness: Maintain detection, containment, communication, remediation, and post-incident review procedures, with lessons converted into controls and tests.