Unit 6: Responsible AI, Security, and Enterprise Governance - Subjective Questions

Responsible AI is the practice of designing, developing, deploying, and operating AI systems so that they are ethical, lawful, safe, transparent, and aligned with human values.

Major principles include:

• Fairness: AI should not systematically discriminate against individuals or groups.
• Reliability and safety: The system should perform consistently, handle failures safely, and remain within defined operating boundaries.
• Privacy and security: Personal and confidential information must be protected throughout the AI lifecycle.
• Transparency: Users should be informed when AI is being used and should understand its purpose and limitations.
• Explainability: Important outputs should be supported by understandable reasons or evidence.
• Accountability: Clearly identified people or teams must be responsible for AI decisions and outcomes.
• Inclusiveness: Systems should accommodate diverse users, abilities, languages, and contexts.
• Human oversight: High-impact decisions should include suitable human review, intervention, and appeal mechanisms.

An ethical AI system requires controls across its complete lifecycle:

1. Problem definition: Confirm that AI is appropriate for the use case and identify affected stakeholders.
2. Impact assessment: Evaluate possible harms involving discrimination, privacy, safety, employment, and accessibility.
3. Responsible data management: Verify consent, relevance, quality, representativeness, ownership, and lawful use of data.
4. Model development: Test for bias, robustness, explainability, security, and unacceptable behavior.
5. Human oversight: Define decisions that require human approval and provide escalation or appeal routes.
6. Deployment review: Conduct technical, legal, security, and ethics approvals before release.
7. Continuous monitoring: Observe outputs, complaints, drift, failures, and impacts on different user groups.
8. Remediation: Correct harmful behavior, notify affected parties when necessary, and document lessons learned.

Ethical AI is therefore not a one-time model property; it is an ongoing organizational process supported by governance and accountability.

• AI governance controls how AI systems are selected, developed, validated, deployed, monitored, and retired. It addresses model risk, fairness, explainability, autonomous actions, human oversight, and AI accountability.
• Data governance manages data ownership, quality, classification, lineage, retention, access, consent, and privacy. It ensures that data used by AI is accurate, lawful, traceable, and appropriately protected.
• IT governance aligns general technology investments and operations with business objectives. It covers infrastructure, service management, cybersecurity, architecture, continuity, and technology spending.

The three areas overlap. For example, a generative AI service depends on governed training data, secure IT infrastructure, and AI-specific controls over prompts, outputs, models, tools, and autonomous behavior. Effective enterprise governance integrates all three rather than treating them as isolated functions.

A suitable enterprise AI governance model can use multiple layers:

• Board or executive committee: Defines the organization's AI risk appetite and approves high-impact strategy.
• AI governance council: Establishes policies, prohibited uses, control requirements, and risk classification standards.
• Responsible AI or ethics committee: Reviews fairness, human rights, social impact, transparency, and controversial use cases.
• Risk, legal, privacy, and security teams: Assess regulatory, contractual, privacy, operational, and cyber risks.
• Model owners: Remain accountable for the business purpose, performance, controls, and lifecycle of each system.
• Independent validators: Test systems separately from the development team.
• Operations teams: Monitor production behavior, incidents, costs, drift, and service reliability.
• Internal audit: Independently evaluates whether policies and controls are effective.

The model should include an AI inventory, risk tiers, approval gates, documented exceptions, periodic reviews, incident escalation, and clear responsibility through a framework such as a RACI matrix.

Important AI safety mechanisms include:

• Input validation: Reject malformed, unsafe, or unauthorized requests.
• Content filtering: Detect harmful inputs and outputs, including violence, abuse, data leakage, or prohibited instructions.
• Guardrails: Restrict the topics, tools, data sources, and actions available to the agent.
• Least privilege: Give the agent only the permissions needed for the current task.
• Sandboxing: Execute code or risky operations in isolated environments.
• Human approval: Require confirmation before financial, legal, destructive, or high-impact actions.
• Rate and spending limits: Prevent loops, denial-of-service behavior, and uncontrolled resource consumption.
• Timeouts and action limits: Restrict the duration and number of autonomous steps.
• Fail-safe behavior: Stop or move to a safe state when confidence is low or controls fail.
• Logging and monitoring: Record decisions, tool calls, policy violations, and outcomes.
• Kill switch: Allow authorized personnel to suspend the system quickly during an incident.

Safety should use defense in depth, because no single mechanism can prevent every failure.

A prompt injection attack attempts to manipulate a language model or AI agent into ignoring trusted instructions, revealing protected information, or performing unauthorized actions.

• Direct prompt injection: The attacker places malicious instructions directly in the user prompt. For example, a user may ask the model to ignore its security policy and reveal its hidden system instructions.
• Indirect prompt injection: Malicious instructions are embedded in external content that the agent later reads, such as a website, document, email, image, or retrieved knowledge record. For example, a webpage may contain hidden text instructing a browsing agent to send confidential data to an attacker-controlled service.

Indirect injection is especially dangerous for agentic systems because they can retrieve untrusted content and invoke tools. The model should therefore treat external content as data rather than authority, preserve instruction priority, restrict tool permissions, and require approval for sensitive actions.

A layered prompt injection defense should include:

1. Separate instructions from data: Clearly delimit trusted policies, user input, and retrieved content.
2. Enforce instruction hierarchy: System and enterprise policies must not be overridden by untrusted content.
3. Validate and sanitize inputs: Detect suspicious phrases, hidden content, encoded payloads, and unexpected formats.
4. Constrain retrieval: Use approved sources, access-aware retrieval, and metadata-based filtering.
5. Minimize privileges: Use short-lived, task-specific credentials and narrowly scoped tool permissions.
6. Validate tool calls: Check parameters, destination, authorization, and business rules before execution.
7. Require human approval: Apply it to payments, account changes, deletion, disclosure, and other high-impact actions.
8. Filter outputs: Prevent secrets, personal data, system prompts, and unsafe content from being returned.
9. Use secure sandboxes: Isolate code execution, browsing, and file processing.
10. Test continuously: Conduct adversarial testing using direct, indirect, multilingual, and encoded attacks.
11. Monitor incidents: Log injection attempts and automatically suspend abnormal sessions.

Because language models cannot reliably identify every malicious instruction, deterministic authorization outside the model must control sensitive actions.

• Authentication verifies the identity of a user, application, service, or agent. Common methods include passwords, multi-factor authentication, certificates, tokens, managed identities, and workload identities.
• Authorization determines what the authenticated identity is permitted to access or perform. Common approaches include role-based access control, attribute-based access control, policy-based access control, and resource-level permissions.

For example, an employee may authenticate using Microsoft Entra ID. Authorization policies may then allow the employee to query an internal AI assistant but prevent access to confidential legal documents. Similarly, an AI agent may authenticate through a managed identity but receive permission only to read a specific database and not delete records.

Both are required: authentication without authorization can grant excessive access, while authorization cannot be applied reliably without a verified identity.

• Least privilege gives each user, service, or agent only the minimum permissions needed for a task. This reduces damage if the agent is manipulated or compromised.
• Role-based access control, or RBAC, assigns permissions according to job or service roles. For example, a support agent may read customer cases but may not modify payment records.
• Zero trust follows the principle of never trust, always verify. Every identity, device, network request, and action is continuously evaluated rather than trusted because it is inside a corporate network.

In agentic AI, these controls should be implemented through scoped identities, short-lived tokens, conditional access, network restrictions, step-up authentication, tool-specific permissions, and repeated authorization checks. Permissions should be evaluated at the moment of action rather than assumed from the original prompt. This limits lateral movement, privilege escalation, and unauthorized tool use.

Important Azure security practices include:

• Use Microsoft Entra ID for centralized identity, multi-factor authentication, conditional access, and workload identities.
• Apply Azure RBAC with narrowly scoped roles and periodic access reviews.
• Prefer managed identities over stored usernames, passwords, or long-lived API keys.
• Store secrets, certificates, and keys in Azure Key Vault and rotate them regularly.
• Use private endpoints, virtual networks, network security groups, and firewall rules to reduce public exposure.
• Encrypt data at rest and in transit, using customer-managed keys where required.
• Use Azure Policy to enforce allowed regions, encryption, tagging, logging, and approved resource types.
• Enable Microsoft Defender for Cloud, vulnerability management, and security posture monitoring.
• Collect audit and diagnostic logs using Azure Monitor, Log Analytics, and Microsoft Sentinel.
• Secure the software supply chain with code scanning, dependency scanning, signed artifacts, and protected CI/CD pipelines.
• Separate development, testing, and production subscriptions or resource groups.
• Establish backup, recovery, incident response, and key-compromise procedures.

Compliance ensures that AI systems follow applicable laws, regulations, contracts, standards, and internal policies. Auditing provides independent evidence that required controls are designed properly and operating effectively.

Auditors may require:

• An inventory of models, agents, datasets, tools, owners, and deployment environments.
• AI impact assessments and documented risk classifications.
• Data sources, consent records, retention schedules, and data lineage.
• Model cards, system cards, evaluation reports, and known limitations.
• Fairness, privacy, security, robustness, and red-team test results.
• Approval records, policy exceptions, and human oversight procedures.
• Access-control configurations and periodic access reviews.
• Version histories for prompts, models, code, tools, and policies.
• Production logs covering prompts, outputs, decisions, and tool actions, subject to privacy controls.
• Incident records, corrective actions, and evidence of control improvements.

Audit records should be accurate, protected from tampering, retained for an approved period, and accessible only to authorized personnel.

A secure architecture can contain the following layers:

1. Client layer: Users connect through a trusted application using encrypted communication.
2. Identity layer: Microsoft Entra ID authenticates users and enforces multi-factor and conditional access policies.
3. Gateway layer: An API gateway performs rate limiting, request validation, threat detection, and policy enforcement.
4. Application layer: The orchestrator runs in a protected compute environment and applies system instructions and guardrails.
5. AI service layer: Model endpoints are reached through private networking rather than unrestricted public access.
6. Retrieval layer: The search index uses document-level access controls, approved data sources, and tenant filtering.
7. Tool layer: Each tool is exposed through a validated interface with its own identity and least-privilege permissions.
8. Data layer: Databases and storage use encryption, classification, private endpoints, backups, and retention controls.
9. Secrets layer: Credentials and keys are stored in a managed vault and accessed through managed identities.
10. Monitoring layer: Centralized logs, metrics, traces, and security alerts feed a security operations platform.

Sensitive tool actions should require deterministic policy checks and human approval. Development, testing, and production environments must also be isolated.

AI risk assessment begins by identifying assets, stakeholders, threats, vulnerabilities, failure modes, and possible harms. Risks may involve security, privacy, bias, misinformation, unsafe actions, legal violations, financial loss, or reputational damage.

A simple risk score can be expressed as:

where is the likelihood of an event and is its impact. Additional factors such as exposure, detectability, and number of affected users may also be included.

The mitigation process is:

1. Identify risks for each lifecycle stage and use case.
2. Analyze causes, likelihood, impact, affected groups, and existing controls.
3. Prioritize risks according to the organization's risk appetite.
4. Treat each risk by avoiding, reducing, transferring, or formally accepting it.
5. Assign a named owner and completion deadline.
6. Validate that controls reduce the risk as intended.
7. Monitor residual risk and reassess after major changes or incidents.

High-impact and irreversible actions require stronger controls, including human approval, restricted autonomy, and rapid shutdown mechanisms.

Production monitoring should cover several categories:

• Quality metrics: Accuracy, relevance, groundedness, completeness, and task success rate.
• Safety metrics: Harmful content, policy violations, prompt injection attempts, and guardrail activations.
• Security metrics: Failed authentication, unusual tool calls, privilege changes, data-exfiltration patterns, and abnormal network activity.
• Fairness metrics: Differences in error rates or outcomes across relevant groups.
• Operational metrics: Latency, availability, token usage, cost, retries, and resource consumption.
• Agent metrics: Number of reasoning steps, repeated loops, failed actions, unauthorized action attempts, and human override frequency.
• Change metrics: Data drift, concept drift, model-version changes, prompt changes, and retrieval-quality degradation.
• User signals: Complaints, corrections, appeals, low ratings, and incident reports.

Thresholds should trigger alerts, throttling, human review, rollback, or automatic suspension. Monitoring must protect privacy by minimizing sensitive log content, controlling access, and applying suitable retention periods.

• Preventive controls attempt to stop incidents before they occur. Examples include authentication, least privilege, input validation, private networking, content filters, secure coding, tool allowlists, and human approval for high-risk actions.
• Detective controls identify incidents or control failures during or after their occurrence. Examples include audit logs, anomaly detection, policy-violation alerts, integrity monitoring, red-team exercises, and behavior analytics.
• Corrective controls reduce harm and restore normal operation after an incident. Examples include credential revocation, session termination, rollback, model quarantine, data restoration, patching, user notification, and policy updates.

A mature AI security program combines all three. Preventive controls reduce attack probability, detective controls shorten the time needed to discover problems, and corrective controls limit impact and support recovery. Relying on only one category creates significant gaps.

Multi-agent systems introduce challenges such as:

• Compromise of one agent influencing other agents.
• Propagation of malicious prompts or false information.
• Confused-deputy attacks in which one agent misuses another agent's privileges.
• Impersonation, message tampering, replay attacks, and unauthorized task delegation.
• Excessive combined permissions across cooperating agents.
• Unclear accountability when decisions emerge from several agents.
• Cascading loops, resource exhaustion, and uncontrolled coordination.

Agent communication can be secured by:

• Assigning every agent a unique, verifiable workload identity.
• Encrypting messages in transit and authenticating both endpoints.
• Digitally signing messages or using integrity-protected tokens.
• Applying message schemas, timestamps, expiration periods, and replay protection.
• Authorizing every delegated task and limiting delegation depth.
• Separating trust domains and sandboxing agents with different risk levels.
• Recording message provenance, tool calls, and decision traces.
• Using a policy-enforcement gateway to validate inter-agent requests.
• Requiring human approval for high-impact workflows.

Agents should not trust another agent merely because it belongs to the same application.

• Human-in-the-loop: A person must review or approve an AI recommendation before the action occurs. This is appropriate for high-impact, ambiguous, or irreversible decisions.
• Human-on-the-loop: The AI operates autonomously, but a person monitors it and can intervene, override, or stop it. This is suitable when rapid automation is needed but meaningful supervision remains possible.
• Human-out-of-the-loop: The AI operates without routine human review. This should generally be limited to low-risk, reversible, well-tested, and tightly bounded tasks.

The correct approach depends on impact, reversibility, confidence, speed, legal duties, and the organization's risk appetite. Meaningful oversight requires trained reviewers, sufficient context, manageable workloads, clear escalation paths, and actual authority to stop or reverse the system's action.

AI systems may process personal, confidential, regulated, or proprietary information. Poor data handling can cause privacy violations, identity theft, discrimination, legal penalties, and loss of trust.

Suitable controls include:

• Purpose limitation: Use information only for approved and communicated purposes.
• Data minimization: Collect and transmit only the information required for the task.
• Consent and lawful basis: Confirm that processing is legally permitted.
• Classification and labeling: Identify sensitive data and apply stronger controls accordingly.
• De-identification: Use masking, tokenization, pseudonymization, or anonymization where appropriate.
• Encryption: Protect data at rest and in transit.
• Access control: Restrict data according to identity, role, purpose, and document-level permissions.
• Retention and deletion: Keep information only for an approved period and support deletion requests.
• Output filtering: Prevent the model from exposing personal data, credentials, or confidential records.
• Privacy assessment: Evaluate privacy risks before deployment and after major system changes.

Prompts and outputs should not automatically be stored or reused for model training without explicit governance approval.

An effective AI incident response plan should include:

1. Preparation: Define incident categories, severity levels, contacts, tools, evidence requirements, and communication templates.
2. Detection and reporting: Use monitoring, user reports, security alerts, and audit reviews to identify harmful or abnormal behavior.
3. Triage: Determine the affected models, agents, users, data, tools, business processes, and legal obligations.
4. Containment: Disable the agent, revoke credentials, block malicious prompts, isolate resources, limit tools, or switch to a safe fallback.
5. Preservation of evidence: Secure logs, prompts, outputs, model versions, configuration, and tool-call histories while protecting privacy.
6. Eradication: Remove malicious content, repair vulnerabilities, update policies, rotate secrets, and correct affected data.
7. Recovery: Restore services gradually, validate controls, monitor closely, and communicate with stakeholders.
8. Post-incident review: Identify root causes, assign corrective actions, update tests and training, and reassess residual risk.

The plan should define when regulators, customers, law enforcement, or affected individuals must be notified.

A high-risk AI agent should be approved only after the following checks:

• Business and ethics: The purpose is legitimate, necessary, proportionate, and aligned with organizational values.
• Ownership: A business owner, technical owner, data owner, and risk owner are identified.
• Risk classification: Security, privacy, fairness, safety, legal, operational, and reputational impacts are documented.
• Data governance: Data sources, quality, consent, lineage, retention, and access rights are verified.
• Model evaluation: Accuracy, robustness, groundedness, bias, explainability, and limitations are documented.
• Security testing: Prompt injection, data leakage, privilege escalation, adversarial inputs, and tool misuse are tested.
• Access control: Authentication, least privilege, segregation of duties, and credential rotation are implemented.
• Agent boundaries: Allowed tools, actions, destinations, spending, execution time, and delegation are explicitly limited.
• Human oversight: Approval points, appeal routes, overrides, and emergency shutdown procedures are established.
• Architecture: Private connectivity, encryption, secret management, isolation, backups, and recovery are validated.
• Compliance: Legal, regulatory, contractual, privacy, and audit obligations are satisfied.
• Monitoring: Metrics, logs, alerts, review schedules, rollback criteria, and incident response procedures are ready.
• Final approval: Independent reviewers sign off, residual risks are formally accepted, and deployment is recorded in the AI inventory.