Unit 6: Evaluation, Safety and Deployment of LLM and Agentic Systems - Subjective Questions

Key evaluation metrics include:

• Task success rate: Percentage of tasks completed according to predefined success criteria.
• Answer quality: Measures correctness, relevance, completeness, coherence, and factual consistency.
• Tool-use accuracy: Evaluates whether the agent selects the correct tools and supplies valid arguments.
• Planning quality: Assesses whether the generated plan is valid, efficient, and logically ordered.
• Latency: Measures the time required to complete a task or produce a response.
• Cost: Includes token consumption, model calls, tool usage, and infrastructure expenses.
• Robustness: Measures performance under noisy, ambiguous, adversarial, or out-of-distribution inputs.
• Safety violation rate: Tracks harmful, restricted, private, or policy-violating outputs.

Task accuracy alone is insufficient because an agent may obtain a correct final answer through unsafe actions, excessive tool calls, fabricated reasoning, or inefficient execution. Evaluation must therefore consider quality, safety, reliability, efficiency, and user impact together.

A safety filter classifies inputs or outputs as either unsafe or safe.

• Precision measures how many items flagged as unsafe are genuinely unsafe:
  
• Recall measures how many genuinely unsafe items are detected:
  
• score is the harmonic mean of precision and recall:
  

Here, represents correctly blocked unsafe content, represents safe content incorrectly blocked, and represents unsafe content that bypasses the filter.

High precision reduces unnecessary refusals, while high recall minimizes harmful content escaping detection. The appropriate trade-off depends on risk: safety-critical applications generally prioritize recall, whereas ordinary productivity systems must also avoid excessive false positives.

Input guardrails inspect data before it reaches the model. They may detect prompt injection, malicious instructions, sensitive information, prohibited requests, or malformed inputs.

Execution guardrails constrain the agent while it plans and acts. Examples include:

• Tool allowlists and deny lists
• Parameter validation
• Least-privilege permissions
• Spending and rate limits
• Sandboxed code execution
• Human approval for high-impact actions

Output guardrails examine the final or intermediate response before release. They detect harmful content, unsupported claims, confidential data, policy violations, or invalid structured output.

These layers provide defense in depth. No single guardrail is completely reliable, so combining controls before, during, and after execution reduces the chance that one failure compromises the system.

Prompt filtering analyzes user prompts before model processing to identify harmful, restricted, adversarial, or irrelevant content. It may use keyword rules, regular expressions, classifiers, embedding similarity, policy models, or combinations of these techniques.

Typical objectives include:

• Blocking prohibited requests
• Detecting jailbreak and prompt-injection attempts
• Redacting personal or confidential information
• Enforcing application-specific input formats
• Assigning risk levels for downstream controls

Limitations:

• Attackers can use obfuscation, encoding, multilingual phrasing, or indirect instructions.
• Keyword filters often produce false positives and false negatives.
• Context-dependent harm may not be visible in an isolated prompt.
• An agent may encounter malicious instructions inside retrieved documents or tool outputs.
• Overly strict filtering can reduce usability.

Prompt filtering should therefore be combined with execution controls, output moderation, access restrictions, monitoring, and regular adversarial testing.

Prompt injection occurs when untrusted content attempts to alter an agent's instructions or make it reveal information, misuse tools, or ignore policies. In retrieval-augmented systems, injected instructions may be hidden in retrieved webpages or documents. In tool-using agents, they may cause unauthorized tool calls or data exfiltration.

A layered defense strategy should include:

1. Instruction hierarchy: Treat system and developer policies as authoritative and retrieved content as untrusted data.
2. Content separation: Clearly delimit instructions, user data, and retrieved evidence.
3. Detection: Scan direct prompts, retrieved documents, and tool responses for injection patterns.
4. Least privilege: Give the agent only the tools and data required for the current task.
5. Tool validation: Validate tool names, arguments, destinations, and expected side effects.
6. Approval gates: Require human confirmation for financial, destructive, external, or irreversible actions.
7. Data controls: Prevent secrets from entering prompts unless essential and authorized.
8. Monitoring: Log suspicious behavior and detect unusual action sequences.
9. Adversarial testing: Continuously test direct, indirect, encoded, and multilingual attacks.

Because detection is imperfect, secure design must limit the damage possible after an injection succeeds.

Explainability is the ability to provide understandable and verifiable information about why an agent selected a decision or action. It supports debugging, accountability, user trust, audits, and regulatory compliance.

Useful techniques that do not expose hidden chain-of-thought include:

• Decision summaries: Concise explanations of the main factors behind an action.
• Evidence citations: Links to documents, records, or observations supporting the result.
• Action traces: Structured logs of tools called, arguments supplied, results received, and state changes.
• Policy references: Identification of the rule or constraint that permitted or blocked an action.
• Confidence and uncertainty indicators: Clear communication of evidential limitations.
• Counterfactual explanations: Statements describing what relevant change would have altered the outcome.
• Feature attribution: Identification of influential input features where appropriate.
• Provenance records: Documentation of models, prompts, data sources, and system versions involved.

A good explanation should be faithful, concise, audience-appropriate, privacy-preserving, and independently auditable.

• Interpretability concerns how directly humans can understand the internal mechanism or relationship between inputs and outputs. Simple models are often more intrinsically interpretable than large neural networks.
• Explainability concerns the production of understandable reasons or evidence for a particular output or action, even when the underlying model is complex.
• Transparency concerns disclosure of relevant system information, such as intended use, limitations, data practices, model versions, tool permissions, and evaluation results.
• Auditability concerns whether an independent reviewer can reconstruct and assess system behavior using logs, version records, policies, approvals, and evidence.

These concepts complement one another but are not interchangeable. An agent may provide a plausible explanation without being transparent about its limitations, and a technically opaque model may still be auditable if its inputs, outputs, actions, policies, and versions are recorded reliably.

Major ethical risks and mitigations include:

• Bias and discrimination: Evaluate outcomes across demographic groups, improve dataset coverage, involve domain experts, and provide appeal mechanisms.
• Privacy violations: Apply data minimization, consent controls, encryption, redaction, retention limits, and access management.
• Misinformation: Use grounding, citations, uncertainty communication, fact verification, and human review.
• Loss of autonomy: Disclose AI involvement and allow users to reject, correct, or override decisions.
• Accountability gaps: Define ownership, escalation procedures, audit trails, and incident-response duties.
• Unsafe autonomous actions: Apply least privilege, action limits, sandboxing, and human approval for consequential operations.
• Intellectual-property concerns: Track data provenance, respect licenses, and establish content-handling policies.
• Unequal access and social impact: Evaluate accessibility, affected communities, labor effects, and distribution of benefits and harms.
• Environmental impact: Measure resource usage and optimize model size, batching, caching, and infrastructure efficiency.

Responsible deployment requires continuous stakeholder engagement and risk management across the entire system lifecycle.

Responsible AI is commonly based on fairness, safety, privacy, transparency, accountability, human oversight, robustness, and inclusiveness.

Application across the lifecycle includes:

1. Problem definition: Confirm that AI is appropriate and identify affected stakeholders and potential harms.
2. Data preparation: Establish consent, provenance, quality, representativeness, security, and retention policies.
3. Design: Use least privilege, safe defaults, accessible interfaces, and explicit escalation paths.
4. Development: Document prompts, tools, models, dependencies, and known limitations.
5. Evaluation: Test quality, bias, privacy, security, robustness, and safety under realistic conditions.
6. Deployment: Use staged rollout, version control, approvals, rollback mechanisms, and clear user disclosures.
7. Operation: Monitor performance, safety events, drift, cost, and user complaints.
8. Retirement: Revoke access, archive required evidence, delete data according to policy, and communicate deprecation.

Responsible AI is therefore a continuous governance practice rather than a one-time compliance test.

A comprehensive strategy uses several complementary testing levels:

• Unit tests: Verify deterministic components such as parsers, policy rules, prompt templates, and tool wrappers.
• Contract tests: Confirm that tool calls and structured outputs match required schemas.
• Integration tests: Check interactions among the model, retrieval system, memory, tools, and external services.
• End-to-end tests: Evaluate complete user tasks in realistic environments.
• Regression tests: Ensure model, prompt, or tool updates do not degrade established behavior.
• Safety tests: Test harmful prompts, data leakage, policy violations, and unauthorized actions.
• Adversarial tests: Simulate jailbreaks, prompt injection, poisoned retrieval content, and tool manipulation.
• Load and resilience tests: Measure throughput, latency, timeout behavior, failover, and recovery.
• Human evaluation: Assess usefulness, correctness, clarity, and context-sensitive risks.
• Shadow and canary tests: Evaluate new versions with real traffic while limiting exposure.

Test datasets should include normal cases, edge cases, failure cases, multilingual inputs, and representative production scenarios. Results must be reproducible through versioned models, prompts, tools, datasets, and configurations.

Offline evaluation uses fixed datasets, simulations, recorded traces, or benchmark tasks before or outside live deployment. It is reproducible, comparatively safe, and useful for regression testing. However, it may not capture changing user behavior, real tool failures, or production distributions.

Online evaluation measures performance with live or production-like traffic. Methods include A/B testing, canary releases, user feedback, shadow deployment, and operational telemetry. It reveals realistic behavior but introduces user risk, confounding factors, and privacy concerns.

A sound process combines both approaches:

1. Require new versions to pass offline quality and safety gates.
2. Run them in shadow mode where practical.
3. Release to a small canary population.
4. Compare quality, safety, latency, cost, and user-impact metrics.
5. Expand gradually or roll back when thresholds are violated.

Offline evaluation supports controlled comparison, while online evaluation validates real-world effectiveness.

Red teaming is a structured effort to discover how an AI system can be manipulated, misused, or caused to fail under adversarial conditions.

A red-team exercise should:

1. Define scope and threat model: Identify assets, users, tools, trust boundaries, attackers, and unacceptable outcomes.
2. Create attack categories: Include jailbreaks, prompt injection, data leakage, hallucination, bias, unsafe tool use, resource exhaustion, and privilege escalation.
3. Prepare test environments: Use isolated systems and synthetic data to prevent real-world harm.
4. Execute diverse attacks: Test linguistic variations, encodings, multi-turn strategies, poisoned documents, and chained tool actions.
5. Record evidence: Capture prompts, outputs, tool calls, model versions, severity, and reproducibility information.
6. Prioritize findings: Rank issues by likelihood, impact, exploitability, and detectability.
7. Remediate and retest: Improve policies, prompts, permissions, filters, or architecture and verify the fixes.
8. Convert failures into regression tests: Preserve lessons for future releases.

Red teaming supplements normal testing by deliberately exploring hostile and unexpected behavior.

A scalable serving framework typically contains an API gateway, authentication and rate limiting, request routing, prompt construction, model-serving workers, retrieval and tool services, caching, queues, observability, and safety controls.

Desirable features include:

• Dynamic batching to process compatible requests together
• Autoscaling based on queue depth, throughput, latency, and accelerator utilization
• Load balancing across model replicas
• Streaming for responsive token delivery
• Model parallelism and quantization for large-model efficiency
• Prefix or key-value caching to reduce repeated computation
• Request prioritization and admission control
• Timeouts, retries, circuit breakers, and backpressure
• Multi-model routing based on quality, cost, risk, or task type
• Versioned deployment and rollback
• Distributed tracing and metrics collection
• Tenant isolation and secure secret handling

The framework must balance throughput, latency, availability, cost, safety, and output quality rather than optimizing only raw token generation.

• Vertical scaling adds more compute or memory to one server. It is simple but limited by hardware capacity and may create a large failure domain.
• Horizontal scaling adds more serving replicas. It improves throughput and availability but requires load balancing, synchronization, and effective autoscaling.
• Model parallelism distributes one model across multiple accelerators when it cannot fit efficiently on a single device. Tensor and pipeline parallelism are common forms, but communication overhead can increase latency.
• Request batching combines multiple requests into one model execution. It improves accelerator utilization and throughput, although waiting to form batches can increase latency.

These methods are commonly combined. For example, each replica may use several GPUs through model parallelism, multiple replicas may provide horizontal scaling, and each replica may apply continuous batching. The best configuration depends on model size, workload variability, latency objectives, hardware, and cost constraints.

Production monitoring should cover both traditional software behavior and AI-specific risks.

Operational metrics:

• Request rate, error rate, throughput, and availability
• End-to-end latency and time to first token
• Queue depth, timeout frequency, and retry count
• CPU, memory, accelerator utilization, and cache performance

AI and agent metrics:

• Task success and fallback rates
• Token usage and cost per task
• Hallucination or unsupported-claim indicators
• Safety-filter activations and policy violations
• Tool-selection accuracy and tool failures
• Number of planning steps and repeated-action loops
• User feedback and escalation frequency

Logs and traces:

• Correlation identifiers and timestamps
• Versioned prompts, models, policies, and tool schemas
• Sanitized inputs and outputs
• Retrieval sources and citations
• Tool calls, validated arguments, outcomes, and approvals
• Safety decisions, failures, and fallback actions

Sensitive data must be minimized, redacted, encrypted, and access-controlled. Alerts should map to service-level objectives and incident procedures rather than merely collecting telemetry.

Availability can be defined as the fraction of valid requests successfully served within an agreed time window:

A success definition should include more than an HTTP status code. It may require valid output, completion before a deadline, and absence of critical policy violations.

Latency is usually summarized using percentiles:

• latency: Median user experience
• latency: Performance for most users under heavier conditions
• latency: Tail behavior affecting the slowest requests
• Time to first token: Delay before streamed output begins
• Inter-token latency: Delay between generated tokens
• End-to-end task latency: Total time across planning, retrieval, tools, and generation

An example objective is: 99.9% monthly availability and end-to-end latency below 4 seconds for standard requests. Error budgets then permit a limited amount of failure:

For a target, the error budget is . Teams should pause risky releases or improve reliability when this budget is consumed too quickly.

• Data drift occurs when production input distributions differ from those used during development or evaluation.
• Behavior drift occurs when system outputs or action patterns change because of prompt, model, retrieval, tool, or environmental changes.
• Performance drift is a decline in quality, safety, latency, or cost metrics over time.
• Model drift may refer to changed behavior after a provider updates a hosted model or after an internal model is replaced.

Drift can be detected through distribution comparisons, embedding-based analysis, sampled human review, benchmark replay, safety metrics, tool-use statistics, user feedback, and comparison with baseline or control versions.

Management measures include:

• Version-pinning models, prompts, policies, datasets, and tools
• Defining acceptable metric thresholds
• Triggering alerts when changes are statistically or operationally significant
• Re-evaluating on recent representative samples
• Using shadow, canary, and rollback mechanisms
• Updating tests and reference datasets carefully
• Investigating whether changes originate from users, data sources, models, tools, or infrastructure

Drift monitoring should preserve privacy and distinguish harmless seasonal variation from meaningful degradation.

Reliable deployment in a high-impact application should follow these principles:

1. Clearly bounded purpose: Define supported tasks, prohibited uses, and operational limits.
2. Risk-based autonomy: Reduce autonomy as potential harm, uncertainty, or irreversibility increases.
3. Human oversight: Require qualified review for consequential decisions and offer escalation and appeal.
4. Least privilege: Restrict data access, tools, credentials, and action scope.
5. Defense in depth: Combine input, execution, output, infrastructure, and organizational controls.
6. Evidence-based evaluation: Test against representative quality, fairness, security, safety, and robustness requirements.
7. Safe failure: Use timeouts, action limits, fallbacks, circuit breakers, and fail-closed behavior where appropriate.
8. Observability and auditability: Record sufficient sanitized evidence to reconstruct decisions and incidents.
9. Controlled release: Use shadow evaluation, canaries, feature flags, staged expansion, and rapid rollback.
10. Lifecycle governance: Assign accountable owners and continuously monitor, review, update, and eventually retire the system.

Reliability is a property of the complete socio-technical system, not merely the underlying language model.

Human-in-the-loop control assigns people a meaningful role in reviewing, correcting, or overriding an agent. It is especially important when decisions affect rights, safety, finances, employment, healthcare, or other high-impact areas.

Approval gates pause execution before sensitive actions such as transferring funds, deleting data, sending external communications, changing permissions, or executing code. The reviewer should receive the proposed action, evidence, expected impact, uncertainty, and alternatives.

Fallback mechanisms provide safe behavior when the model, tool, or infrastructure fails. Examples include:

• Asking the user for clarification
• Returning a bounded refusal
• Switching to a simpler verified workflow
• Escalating to a trained operator
• Reverting to a previous stable model
• Queuing work for later processing
• Entering read-only or degraded mode

Human oversight must not be merely symbolic. Reviewers need adequate expertise, time, authority, and understandable evidence. Automation bias should be reduced by displaying uncertainty and making rejection or correction straightforward.

An end-to-end workflow can be organized as follows:

Before deployment:

1. Define intended use, risk level, owners, service objectives, and prohibited outcomes.
2. Build representative quality, safety, security, bias, and resilience evaluations.
3. Threat-model the model, retrieval pipeline, tools, data stores, and external integrations.
4. Configure least-privilege access, approval gates, budgets, action limits, and safe defaults.
5. Version models, prompts, policies, tools, retrieval indexes, and evaluation datasets.

Release:

1. Validate in an isolated staging environment.
2. Run shadow traffic and compare results with the production baseline.
3. Deploy to a small canary population using feature flags.
4. Monitor quality, violations, latency, errors, cost, and user reports.
5. Expand only when predefined acceptance thresholds are satisfied.

Incident response:

1. Detect and classify the incident by impact and urgency.
2. Contain it by disabling risky tools, limiting traffic, rotating credentials, or rolling back.
3. Preserve privacy-safe evidence, including traces, versions, and affected actions.
4. Notify accountable teams and affected parties as required.
5. Remove the root cause and verify remediation with targeted tests.
6. Restore service gradually under enhanced monitoring.
7. Conduct a blameless post-incident review.
8. Add the failure to regression suites and update controls, documentation, and training.

This workflow makes deployment reversible, observable, accountable, and progressively validated.