Unit 5: Testing, Monitoring, and Deployment of AI Agents - Practice Quiz

1 What is the main purpose of testing an AI agent workflow?

2 Which type of test checks a single function or component in isolation?

3 Which resource is most useful for tracing the sequence of actions taken by an AI agent?

4 What does reproducing an error mean during debugging?

5 What does observability help developers understand about an AI agent?

6 Which metric directly measures how long an AI agent takes to respond?

7 What is telemetry in an AI agent system?

8 Which telemetry item helps connect events belonging to one request?

9 What is an AI hallucination?

10 Which method can help detect hallucinations in a factual response?

11 What is output validation used for in an AI agent?

12 Which validation method is appropriate when an agent must return valid JSON?

13 Which deployment strategy sends a new version to a small group of users first?

14 Which Azure service is commonly used to deploy and manage containerized applications?

15 Why is an AI agent commonly exposed through an API?

16 Which HTTP method is commonly used to submit a prompt in the body of an API request?

17 What is a common purpose of continuous integration in an AI agent project?

18 What does horizontal scaling mean for an AI agent service?

19 How can caching improve the performance of an AI agent?

20 Which feature is essential for a production-ready AI agent system?

21 An AI agent uses a language model, a search tool, and a payment API. During automated testing, the team wants repeatable results without making real payments. Which approach is most appropriate?

22 A support agent sometimes sends an incorrect tool parameter after several reasoning steps. What debugging technique would best identify where the incorrect value was introduced?

23 A production agent meets its average latency target, but some users report very slow responses. Which metric is most useful for investigating this issue?

24 An agent request passes through an API gateway, an orchestrator, and three tools. Which telemetry field most directly helps connect all events belonging to the same request?

25 A retrieval-augmented agent must answer only from approved policy documents. Which check is most effective for detecting unsupported claims?

26 A team updates the prompt used by an invoice-processing agent. How should it determine whether extraction accuracy has improved?

27 A company wants to release a new agent version to a small percentage of production traffic before a full rollout. Which deployment strategy should it use?

28 An AI agent deployed on Azure must access Azure Key Vault without storing credentials in source code or environment files. Which option is most appropriate?

29 A client may retry an agent API request after a network timeout. The request can create a purchase order, so duplicate execution must be prevented. What should the API support?

30 A team wants to prevent agent versions with poor answer quality from reaching production. Which CI/CD control best addresses this requirement?

31 Traffic to an AI agent varies sharply throughout the day, and each request can trigger a long-running tool operation. Which architecture best handles these bursts?

32 An agent repeatedly generates embeddings for an unchanged collection of documents, increasing cost and latency. What is the best optimization?

33 A production agent depends on a third-party search service that occasionally fails. Which design most improves resilience?

34 An order-management agent must never call the shipping tool before payment is confirmed. Which test best validates this requirement across many generated inputs?

35 An agent states that a contract ends on June 30, but the retrieved contract passage states July 30. Which automated method is best suited to flag this response?

36 After a new release, tool-call success remains high, but users increasingly abandon conversations. Which additional signal would best help detect the underlying quality problem?

37 A healthcare agent's traces may contain patient information. How should the team collect useful telemetry while reducing privacy risk?

38 A team maintains two identical production environments and wants near-instant rollback if a new agent version fails. Which strategy best fits this goal?

39 A conversational agent stores session history in each application server's local memory. After horizontal scaling, users sometimes lose context. What is the best correction?

40 An agent performs ten independent retrieval requests sequentially, and each request takes about ms. Which change is most likely to reduce total retrieval latency?

41 An AI agent uses a stochastic planner and three external tools. The same input may produce different valid execution paths. Which test design best detects workflow regressions without requiring an exact trace match?

42 A production agent intermittently selects an expensive tool even though an equivalent cached result exists. Logs contain prompts and final answers but not intermediate state. Which change most directly enables deterministic diagnosis?

43 An agent request invokes a planner, retrieval service, and multiple tools across separate processes. Operators must identify which component caused a latency spike for one request. Which observability design is most appropriate?

44 A team adds the complete user prompt as a metric label to diagnose failures. Monitoring costs and query latency rise sharply, and sensitive data may be exposed. What is the best redesign?

45 A retrieval-augmented agent answers questions using documents that may contain outdated or contradictory claims. Which mechanism most reliably detects unsupported statements in the generated answer?

46 Two stochastic agent versions are evaluated on the same 1,000 tasks. Task difficulty varies substantially, and each version is run once per task using matched seeds. Which analysis best estimates whether the new version improved task success?

47 A new agent release changes both prompt logic and the schema of persisted conversation state. A canary deployment routes 5% of traffic to the new release, but users may switch between versions on later turns. Which strategy best prevents state incompatibility?

48 An agent hosted on Azure Container Apps must access Azure Key Vault and Azure AI Search without storing credentials in code or deployment variables. Which configuration best satisfies this requirement?

49 An agent API starts long-running jobs and clients frequently retry after network timeouts. Duplicate retries currently trigger repeated tool actions, including duplicate purchases. Which API contract most directly mitigates this failure?

50 A pull request changes an agent's system prompt. Unit tests pass, but offline evaluations show a 2% quality decrease with lower token cost. The release policy permits at most a 1% quality decrease. What should the pipeline do?

51 A horizontally scaled agent service stores active conversation memory in each instance's local RAM. Autoscaling and load balancing cause later turns to lose context. Which architectural change best supports reliable scale-out?

52 An agent executes retrieval in 120 ms, planning in 300 ms, two independent tool calls in parallel taking 500 ms and 700 ms, synthesis in 250 ms, and fixed orchestration overhead of 80 ms. Assuming these stages are otherwise sequential, what is the end-to-end latency?

53 A financial agent can initiate irreversible transfers through a tool. Which control set best reduces the risk of erroneous autonomous actions while preserving useful automation?

54 A scheduling agent should preserve an existing meeting's duration when the user asks to move it to another start time. Which metamorphic test is most appropriate?

55 Two tools execute concurrently and both update shared agent state. A rare failure occurs when one update overwrites the other. Which diagnostic approach is most likely to reveal the defect?

56 An agent's immediate operational metrics remain stable, but confirmed task-success labels arrive seven days later and show degradation after a knowledge-base update. Which monitoring design would detect this pattern most effectively?

57 A failure occurs in only 0.05% of agent traces. Uniform 1% head sampling rarely captures it, but retaining every trace is too expensive. Which sampling strategy best preserves diagnostic traces under a storage budget?

58 A model assigns high probability to a medical statement, and five sampled responses repeat it consistently, but the retrieved sources do not support it. How should a hallucination detector classify the statement?

59 An Azure-hosted agent must survive a regional outage while preventing the same queued tool command from producing duplicate side effects after failover. Which design is most appropriate?

60 A streaming agent API generates tokens faster than some clients can consume them, causing server buffers to grow until instances run out of memory. Which mechanism best addresses the problem?