Chapter 3 — AI Risk Program Management — Part C: AI Controls Management

On this page

3.10AI Control Types and Control Frameworks
3.11AI Control Selection and Validation
3.12Control Performance
3.13Controls Specific to AI Solutions
3.14Use of AI in Control Management

Part C: AI Controls Management

Controls for AI assets and systems are essential to ensure their proper development, training, deployment, operation, and decommissioning. Controls span governance, technical, procedural, personnel, physical, and ethical measures, ensuring the security, reliability, compliance, and responsible use of AI systems. Controls may cross multiple organizational areas.

Existing technical and nontechnical controls may need to be expanded, and new controls may need to be introduced with the use of AI. The enterprise’s established control framework will need to be updated to account for AI concerns. AI-specific controls may be similar to privacy and cybersecurity controls, with coordination, technical configuration, monitoring and auditing, and upgrading controls combined to contribute to the holistic effectiveness and sustainability of protection measures across several domains.

As with cybersecurity, no single area is responsible for AI controls. Behavior is guided through policy, standards, and procedures. Technical architectures and requirements are designed to prevent or reduce the impact of a threat being realized. Monitoring and continuous improvement will identify negative or unwanted events and adjust to minimize or remove the impact of them.

3.10 AI Control Types and Control Frameworks

Common types of AI controls are listed in figure 3.15.

Figure 3.15—Common Types of AI Controls

Control Category	Controls	Explanation
Governance and Organizational	Accountability (e.g., RACI chart, accountability matrix)	Develop terms of reference or a charter and include roles and responsibilities for artificial intelligence (AI) management and oversight and cross-functional membership of the organization. There should be clear, documented ownership for each AI asset and that asset’s defined outcome(s). Supporting stakeholders include development, operations, audit and assessment, and management and oversight.
	Executive-level role	Appoint a chief data officer (CDO), chief AI officer (CAIO), or other function as the accountable manager and committee head.
	AI policy	Develop AI principles or acceptable use and responsible use policies that align with the organization’s values, culture, and mission. Incorporate regulations and legislation. Include risk management processes. Include asset management processes.
	Standards and frameworks	Several frameworks and standards are available for organizations to implement, providing structure and baseline categories of responsibilities for the development, management, risk management, and oversight of AI systems.
	AI asset management	Creating and maintaining an AI registry with details of all models, including details such as purpose and goals, stakeholders, outcomes, versions, and life cycle management.
	Risk management	An AI risk committee should be created to oversee and report on AI-specific risk areas. Clear ownership and accountability should be assigned for AI risk management and mitigation. Key risk indicators (KRIs) should be established to provide information on the management of AI risk. Internal and external audit teams can provide independent reports on the success of these programs. Vendor risk management should also be included in overall risk management.
	AI legal contracts and agreements	Contract management needs to ensure that vendors and partners meet obligations that support the organization’s adherence to AI laws and regulations. For example, organizations may not have needed to address consent previously when interacting with data but may now need to consider or secure it in order to train AI models.
Technical, Security, and Privacy	Data quality	Document definitions for: Data quality thresholds and rules Assessing data quality and resolving quality issues Monitoring data quality
	Access controls	Existing access controls should be evaluated to ensure they meet new regulations and laws that provide protection for data related to AI products and services. Users (privileged and nonprivileged), systems, applications, and third-party access controls should be adjusted based on the classification of data and the classification of the AI system. Multifactor authentication (MFA) should be implemented. Regular monitoring and reporting of users and certification and recertification of ongoing user access should be implemented.
	Configuration management	For AI development, configurable items are documented, configured based on risk and security requirements, monitored, and patched. Modern environments use automation for configuration management and tightly integrate configuration management into other processes for transparency, security, and incident and problem management. In AI development, configuration management monitoring can provide risk detection and mitigation. For example, a detected vulnerability can be identified and corrected before it can be identified and exploited by an attacker.
	Testing frameworks and protocols	End-to-end testing Scenario analysis Bias and fairness testing
	Privacy	Communication and transport protocols Encryption, hashing, and de-identification Data anonymization Key management Monitoring and logging Identity and access management (IAM) Regular privacy impact assessments (PIAs) that consider regulations (Reporting as a part of governance reporting)
	Data management	Data controls: access controls (including privileged and administrator), encryption, integrity monitoring, database configuration management, backup and resiliency Data architecture Data quality management (including aggregation and integration; data optimization) Data documentation and modeling Data integrity controls Data minimization
	AI incident management	As with other areas of incident response, AI incident response focuses on data breach and operational interference to identify, contain, mitigate, and restore AI processes. Auditors should look at the organization’s incident response plan or data breach response plan to ensure that AI incident response has been considered and is in place for AI products and services.
Operation and Life Cycle Management	Dataflow diagrams (DFDs) and process flow documentation	DFDs include logical, physical, and context flows. These aid stakeholder understanding and explain how data is handled in the AI system and used by the models. Process flows can be multilevel and explain complex processes and subprocesses in areas such as data collection, processing steps, enrichment, and output.
	AI decision-making documentation	Explainability documentation clarifies the “how” (techniques, scores, maps) process from data input to specific output decisions made by the system, including decision-making process documentation (steps, DFDs, process flows) and end user documentation. Model documentation provides details about architecture, algorithms, training data, and performance expectations and measures. Compliance and ethical documentation communicate regulatory requirements, safety, reliability, accountability, and outcomes from assessments of bias and fairness. Data management and traceability documentation accounts for data sourcing and consent, enrichment, and transformations and logs how decisions are made for audit/monitoring.
	Applicable standards	Change management (including updating and patching)
	Version control and version (code) management	Data quality and validation Reproducibility, testing, and training Version management (environment management) Virtualization/containerization management (automated tools) Approval process Deployment or release procedures
	Change management	Documented change process, team or authority, and delegated authority for approvals and emergency changes (formal recordkeeping; retention of change approvals) Risk and potential impact identification (documentation on the ticket or change system) Change management plans (system of record; ticketing or other records; communication of changes; business approval) Change monitoring and reporting (continuous auditing)
	Continuous monitoring and logging	Monitoring of critical systems for AI product delivery and processing Data quality management controls for input validation, data consistency and completeness, and model drift
Systems Development Life Cycle (SDLC)	Project management	SDLC controls depend upon the approach the organization has adopted for development. For example, an organization using DevOps/DevSecOps¹⁶⁷ will approach controls differently than a traditional waterfall organization. For many organizations developing AI, the end product is often never deployed.¹⁶⁸ While this failure rate may not be tied to SDLC methodology, most projects fail due to unclear or incomplete requirements and/or poor leadership. New collaborative relationships can also impact the success of a project.
	Ideation phase	Documentation of the problem space Goals and objectives aligned with business goals and strategic planning outcomes Understanding of potential adverse outcomes or harm
	Design phase	Technical design documentation Data management (data collection: purchased or acquired?; data quality; data optimization; data fitness: is the data complete and representative of the population it will operate on in final deployment [bias mitigation/avoidance]?) Requirements defined and documented (model types to be used)
	Deployment management	Some organizations may use automation to deploy packages once committed. If an organization uses a separate deployment methodology and uses automation to either automatically promote packages or code once it has been committed, this area should be monitored by the operation and audited.
Legal, Compliance, and Regulatory	Risk and impact assessments	Systematically identify, analyze, and manage risk associated with the deployed capabilities of an AI system. Ensure AI systems are categorized so that high-risk systems are managed and assessed according to requirements such as the EU AI Act¹⁶⁹ or New York City’s Local Law 144. Regardless of framework or regulation, AI systems must be iteratively assessed as systems are dynamic, data changes, and risk management must become dynamic to adjust to changes over time. For high-risk AI systems, a FRIA must be completed. A data protection impact assessment (DPIA) is specific to privacy and data protection risk.
	Fundamental rights impact assessment (FRIA)	For organizations that have systems classified as high risk under the EU AI Act, a FRIA must be conducted. These systems are considered to have an impact on human rights, safety, or potentially health. These include:¹⁷⁰ Biometric identification Management and operation of critical infrastructure Education and vocational training Healthcare diagnostic systems Employment, worker management, and access to self-employment Access to and enjoyment of essential private services and public services and benefits Law enforcement Migration, asylum, and border control management Administration of justice and democratic processes
	Legal and regulatory	Global, national, and local laws Industry regulations
	Risk management	Ensuring systems are properly classified (e.g., EU AI Act) and follow requirements set by applicable regulations Adherence to mandatory and voluntary requirements
	AI legal contracts and agreements	Contract management needs to ensure that vendors and partners meet obligations that support the organization’s adherence to AI laws and regulations. For example, organizations may not have needed to address consent previously when interacting with data but may now need to consider or secure it in order to train AI models.
	AI whistleblower process	An organization should have a mechanism that encourages self-reporting of actions and issues that may allow AI products and services to cause harm or a negative societal impact. A whistleblower process commonly allows individuals to report violations of policies or laws, illicit or illegal activities or outcomes, misconduct, or unethical behavior or practices. Organizations may already have a whistleblower process in place, and those that do not may consider establishing a process.
	Regulatory reporting and decision making	AI needs to become a part of existing compliance efforts, and specific AI mandates need to be discussed and communicated from the AI committee. Compliance may entail establishing a voluntary “Code of Practice.”¹⁷¹
Ethical and Human Values	Ethical impact assessment (EIA)	Conduct an EIA for AI systems. Align with available ethical frameworks from industry (e.g., financial, healthcare) or a corporation (e.g., Microsoft,¹⁷² IBM,¹⁷³ Google¹⁷⁴). There are several free or open-source frameworks and guidelines available to enable an organization to become familiar with responsible AI and AI ethics and to embed these concepts into development and operational practices.¹⁷⁵
	Consent management	Ensure the right to opt out. Check opt-out of AI processing/decision making provided to stakeholders. Ensure user engagement and control over personal data use and processing. Validate processes and procedures for documenting and processing the consent status of data for AI model training. Review data retention procedures. Review organizational procedures around the right to be forgotten.
	Bias management	Evaluate analysis of bias and fairness in the testing process and outcomes. Ensure protection against manipulation through development activities and security protections. A regular FRIA can enumerate bias in development and outcomes, such as dignity, discrimination, and inequalities. Check for human bias in the development team, requirements definitions, and testing. Ensure regular bias testing against protected characteristics. Bias management should also include domain-specific risk, such as bias in financial datasets, geographic representation, or linguistic coverage, depending on the AI system’s use case. Validate that processes exist to evaluate training data to ensure that data management controls are implemented and operating as expected. Validate representative data sampling procedures to determine if the inferences or predictions made by the model accurately represent the larger population. For example, it was found that in some facial recognition algorithms error rates for dark-skinned females were as high as 34.7% compared to 0.8% for light-skinned males.¹⁷⁶
	Transparency and explainability	Disclosure of AI decision making Explainability and transparency of decision making (validate documentation of the AI system’s capabilities and limitations; model documentation of design, logic, and decisions; check disclosure of AI use/interaction to stakeholders; validate documentation of training data collection; confirm use of model explainability and interpretability tools:¹⁷⁷ LIME [explains individual predictions], SHAP [feature importance for individual predictions], rule-based models [review decision rules the AI system uses])
	Human oversight	AI systems should include human oversight. Ensure processes exist for human oversight of AI decisions/output. Validate an appeal process for AI decisions. Conduct in-house, independent AI ethical reviews. Evaluate human review of fairness metrics, which should tie into governance and ethics management.

Source: ISACA, ISACA AAIA Official Review Manual, USA, 2025

3.10.1 AI Controls Frameworks

While the field is still evolving, some organizations have published, or are in the process of publishing, AI control frameworks, including:

NIST AI RMF
Cloud Security Alliance (CSA) AI Controls Matrix
MITRE ATLAS Matrix
OWASP GenAI Security Project Top 10 for LLMs and GenAI
ISACA AI Audit Toolkit
IBM GenAI Controls Framework
Google’s Secure AI Framework
SANS Institute Draft Critical AI Security Guidelines

Enterprises can leverage this guidance to find controls that work to mitigate risk in AI systems and tools that are deployed, used, or leveraged.

3.11 AI Control Selection and Validation

As AI solutions are often built on or integrated into existing networks and infrastructure, it is important to evaluate the effectiveness of existing controls to see if they are adequate for addressing the complex needs of AI solutions.

When enterprises purchase AI solutions or select a vendor to provide part or all of an AI solution, careful review of the vendor’s preset security options and controls is necessary to determine if they are adequate and support the enterprise’s risk appetite and tolerance levels. See 3.17 AI Vendor Management for more information.

3.11.1 Control Gap Analysis

An important first step when determining what controls are needed for AI solutions is to identify any gaps in existing controls that are created by the risk presented by the use of AI. This begins with evaluating what controls already exist in the enterprise and where the primary controls are insufficient, ineffective, or infeasible to implement.

Control gaps represent the disparity between the current state of controls and the desired state necessary to maintain risk within acceptable levels. A thorough control gap assessment begins with an accurate understanding of the current control environment, which can be established through various sources such as audits, control tests, incident reports, vulnerability assessments, and direct observations. This assessment provides a baseline for identifying weaknesses or deficiencies in the control framework that may expose the organization to unacceptable AI-related risk. For example, a database of customer information is properly secured by the marketing team, but when the data is shared with the development team, they discover that the new database lacks proper access controls.

Once these gaps are identified, the enterprise can evaluate what adjustments need to be made to existing controls or what controls may need to be developed to address new risk areas not previously addressed by these existing controls.

3.11.2 Alternative and Compensating Controls for AI

When primary controls cannot adequately mitigate risk—due to constraints such as cost, operational requirements, or technical limitations—alternative or compensating controls must be considered. These controls serve to offset the risk that cannot be directly addressed by the primary controls. Effective compensating controls are designed to complement existing controls without duplicating efforts or increasing the risk surface. Examples include layered defense strategies, enhanced supervision, increased audit frequency, and comprehensive logging of AI system activities.

The process of implementing alternative controls requires careful design and selection to ensure they align with organizational risk appetite and operational realities. For AI systems, compensating controls might include additional monitoring of model behavior, stricter access controls on training data, or human oversight mechanisms to review AI outputs when automated controls are limited.

Once alternative controls are identified and implemented, their effectiveness must be validated through testing and continuous monitoring. See Part D: AI Risk Metrics, Monitoring, and Reporting for more information.

3.11.3 Alignment With Risk Management Frameworks

AI control evaluation must be aligned with the organization’s broader risk management frameworks and policies. This alignment ensures that AI controls support enterprise risk appetite and tolerance levels and comply with applicable legal, regulatory, and ethical standards. The evaluation process should verify that AI controls are integrated with existing governance structures, such as risk committees and audit functions, and that accountability for AI risk management is clearly assigned. Furthermore, controls should be assessed for their compliance with AI-specific requirements, including DPIAs, FRIAs, and ethical impact assessments (EIAs), especially for high-risk AI systems.

3.11.4 Validation of AI Controls

Validating AI controls ensures that risk mitigation measures are functioning as intended and that AI systems operate securely, reliably, and ethically. Given the dynamic nature of AI models and data, validation must be iterative and adaptive, incorporating feedback from monitoring and audit findings to continuously improve the control environment. It should also occur during major AI life cycle events (e.g., after model retraining, data updates, or architecture changes) to confirm that control effectiveness remains consistent, and adversarial robustness testing should be conducted to ensure AI systems can withstand malicious inputs and manipulation attempts.

Control Testing

Once controls are selected, they should be tested to ensure they are implemented properly to manage risk to acceptable levels. Testing helps to confirm that controls meet their design objectives and comply with organizational policies and regulatory requirements. Techniques include:

End-to-end testing—Verifies the AI system’s functionality across all components and workflows, ensuring that controls are integrated and effective in real-world scenarios
Scenario analysis—Simulates various operational and threat scenarios to assess control responses and resilience under different conditions
Bias and fairness testing—Evaluates controls designed to detect and mitigate bias, ensuring equitable outcomes and compliance with ethical standards

Continuous Monitoring

Validation requires ongoing monitoring to detect control degradation or emerging risk. Continuous monitoring includes:

Data quality management controls
Access and configuration monitoring
Logging and anomaly detection

Documentation and Evidence Collection

Maintaining detailed documentation of control implementation, testing results, and monitoring activities is essential for validation. This includes:

Records of test cases, scenarios, and outcomes demonstrating control effectiveness
Logs and reports from continuous monitoring systems
Evidence of compliance with data governance, privacy, and security policies

Independent Assurance and Audits

Independent assessments by internal or external auditors provide an objective evaluation of control effectiveness. Auditors validate that controls are implemented as designed and are effective in mitigating AI risk. They also assess whether controls align with relevant frameworks, standards, and regulations.

3.12 Control Performance

Controls can lose effectiveness over time. This is especially true with AI solutions, as model performance can degrade with the introduction of new information. The rapid pace at which AI technologies are advancing along with a threat landscape that changes at the same, or an even quicker, pace emphasizes the need to ensure risk treatment strategies continue to perform as expected. Monitoring performance also enables enterprises to apply compensating controls or change control thresholds given this dynamic environment.

Effective control performance monitoring requires establishing clear processes and procedures aligned with enterprise objectives and risk management frameworks. This includes:

Identifying and confirming control owners and stakeholders responsible for monitoring activities
Engaging stakeholders to communicate monitoring objectives and reporting requirements
Integrating control monitoring with broader IT and enterprise performance management systems
Defining life cycle management and change control processes to maintain monitoring effectiveness
Allocating appropriate resources to support ongoing monitoring efforts

Data for monitoring should be collected from a variety of sources, including operational and security logs, security information and event management (SIEM) systems, network operations centers (NOCs), and testing results. The data must be validated for accuracy and integrity before analysis against performance targets. Advanced tools, including AI-based anomaly detection, can enhance the identification of control failures or unusual behavior indicative of risk.

When monitoring reveals control deficiencies or noncompliance, the risk practitioner should collaborate with control owners to recommend and implement mitigation measures. These include:

Adjusting or enforcing existing controls
Implementing new controls to address gaps
Modifying business processes to reduce risk exposure

Control performance monitoring should be iterative and adaptive, reflecting changes in the AI risk environment, emerging threats, and organizational priorities. Regular self-assessments by control owners foster ownership and accountability, while independent assurance reviews provide objective validation of control effectiveness. All monitoring activities and management acknowledgments should be documented in the risk registry to maintain transparency and support audit readiness.

3.13 Controls Specific to AI Solutions

As noted, AI solutions may use many of the same controls that other IT and network implementations currently use within the enterprise. At the same time, AI solutions can often be more complex; for example, a group of multiple AI agents is deployed to aid a customer service team as the first contact for a retailer. Each of these agents is a separate LLM that needs to be monitored for performance.

Data protection and privacy concerns also play a role in AI risk, and many enterprises may need to deploy more sophisticated controls around this area.

In addition to technology architecture, AI introduces nontechnical risk (see 3.1.2 Nontechnical Threats for more information) that requires implementation of specialized controls.

3.13.1 Controls for AI-related Data Privacy

Common AI-related data privacy controls are listed in figure 3.16.

Figure 3.16—Common AI-related Data Privacy Controls

Control	Description
Data privacy and handling protocols	Implement and document protocols and practices for data handling, ensuring data encryption, anonymization, and access controls.
Artificial intelligence (AI) data retention and encryption protocols	Deploy encryption techniques and define and implement clear data retention timelines to bolster AI data security.
Privacy-first data handling	Ensure models handle data with privacy as a priority, especially when regulatory changes might impact operations.
Differential privacy	Ensure the model does not leak individual training data information.
Privacy enhancing technologies (PETs)	Integrate technologies that enhance user privacy into AI systems, ensuring that data is protected and privacy standards are met or exceeded.

Source: ISACA, Artificial Intelligence Audit Toolkit, link

3.13.2 Controls Related to AI Ethical Considerations

Ethics are a major concern when adopting AI technologies; however, this is an area that is unfamiliar to many enterprises and the IT security teams who are accustomed to focusing on the technical side of an implementation. In general, there are two AI capabilities that make ethical considerations regarding its use different from other technical areas—automatic decision making and self-learning. Both of these AI attributes make it necessary to examine how AI’s autonomy from human involvement may create undue harm or risk to people.

Common ethics controls include:¹⁷⁸

Ensuring human oversight of AI systems (e.g., HITL)
Establishing an AI ethics and compliance committee
Establishing and evaluating the effectiveness of an AI code of conduct
Ensuring and documenting that AI systems adhere to standards of health, safety, laws, environmental considerations, and fundamental rights
Ensuring transparency, fairness, and lack of bias
Evaluating the environmental impact of the use of AI

3.13.3 Safety and Human Oversight

While concerns around how AI may cause physical harm to humans remain hypothetical, there are real considerations when it comes to automation and decision making carried out by AI algorithms and models.

For example, a grocery store uses a robotic machine to clean its floors. What security measures are in place when that machine encounters a shopping cart and a customer? What decisions has the robot been programmed to make on its own? The safety implications are numerous for the autonomy of objects to maneuver in the environment.

This extends to AI systems as well. Technological developments have started to provide AI with agency (e.g., the ability to act independently and make decisions). This has raised some ethical and philosophical questions, but there are more practical issues to resolve:

If AI can act and make decisions autonomously or semiautonomously, how can AI or its human creators be held accountable for the resulting consequences of its decisions and actions and to what extent?
What self-regulation does AI have to act and behave ethically and safely?
What happens if AI makes a decision that conflicts with a user’s values?
What expectations should users have for AI to explain its decisions if designers cannot fully interpret them?

With AI gaining greater agency and autonomy, robust governance and increased supervision over AI outcomes are key. Figure 3.17 describes common controls related to AI supervision and safety.

Figure 3.17—Controls Related to AI Supervision

Control	Description
Logging and monitoring	Deep learning (DL) models are the most complex to understand and explain, largely because of their large dataset sizes and high dimensional calculations. These complex models should be made transparent with detailed logging of their decision-making paths. This practice allows the artificial intelligence (AI) designer to see the input and output of each neural network (NN) path the model took through its “chain of thoughts.” Paired with the data input and resulting inferred output from the model, an audit log is created to aid in debugging and troubleshooting.
AI observability	AI observability practices and tools help ensure the availability, reliability, performance, and trustworthiness of the AI system. Categories of AI observability tools include: Data pipelines—Proactive monitoring of the pipeline used for model training and inference provides insights into how and why the AI solution makes the decisions it does. Observability into these processes allows enterprises to prevent issues such as prompt injection attacks or unacceptable data quality during preprocessing. Monitoring infrastructure and the system—Monitoring of key system health, consumption, and processing telemetry can help organizations identify and anticipate heavy use of an AI system, which could overload system resources and create availability and performance problems. Interpretability—The more complex the AI model (e.g., DL, NN), the more difficult it is to interpret the model. Some strategies to improve interpretability include counterfactual explanations, feature virtualization, and influential instance.
Human in the loop (HITL)	Ensuring a person oversees and monitors AI and makes final decisions related to the AI outputs is key to ensuring the safety and accuracy of AI decisions.

Human in the Loop (HITL)

For critical decisions and actions, another AI supervision strategy is employing a HITL control. For example, before a patient’s final diagnosis of a disease, a human doctor could be placed in the loop to review the AI model’s decision. This puts a human in the final decision-making role to approve the AI workflow. The workflow pipeline of the AI system must be designed and implemented to require a HITL for specified conditions and explicit approval before the workflow is allowed to continue.

The term AI in the loop (AITL) represents a paradigm shift from the traditional HITL approach, emphasizing a collaborative dynamic in which AI systems actively assist and augment human decision-making processes. In this framework, AI serves as a support tool, providing data-driven insights and recommendations, while humans maintain primary control over critical decisions.

3.13.4 Access Controls

While AI is being implemented as an access control itself, protecting access to data used for and created by AI needs additional consideration over typical access control policies.

An AI access control policy should include the authorization, integration, duration, and type of data to be accessed, with appropriate controls to ensure that access is protected for those who would normally not have access without the data being part of an AI model.

For example, a technical supervisor does not have access to employee HR records. However, employee records may be included in an internal AI model for staffing, utilization, etc. The same supervisor may be granted access to the same model for other analytics and model supervision. In this scenario, the supervisor may have access to data in the model that would be restricted in other situations.

Key considerations for AI access control include:

A centralized AI access control policy
Role-based access controls (RBACs)
Validation to test that proper access is provided to mixed data models
Review of access controls after each development release to ensure testers or reviewers no longer have access if they no longer need it
Least privilege

3.13.5 Zero Trust

The concept of zero trust within a traditional security model is fairly clear-cut. Enterprises should never assume security unless verified. This is applicable to AI solutions, particularly regarding the model and the data used. Some traditional security zero trust concepts that can apply to AI include:¹⁷⁹

Identity and access management (IAM)—Access controls are key to ensuring that unauthorized access to the AI model and data associated with it does not occur. Multifactor authentication (MFA) and the principle of least privilege should be applied.
Network segmentation—Network segmentation can ensure secure connections between data sources and the AI model to limit communication from other areas of the network to the AI model.
Data encryption—Encryption is key to safeguard data throughout its life cycle.
Data loss prevention (DLP)—DLP helps to monitor for and prevent data leaks and block unauthorized data processing and transfers. DLP policies ensure that sensitive data is not transmitted to external entities, such as public AI models.
User and entity behavior analytics (UEBA)—UEBA is used to detect unusual and anomalous behavior, which can point to potential attacks on an AI solution. These solutions can be set up to alert security teams to these behaviors.

Zero trust in AI requires additional safeguards for mixed data models (structured, unstructured, synthetic), ensuring that different data types are properly segmented and controlled.

Zero trust in AI means that decisions made by AI solutions are not automatically assumed to be correct.

In order to fully trust that an AI is making “correct” decisions, three key areas must be considered:¹⁸⁰

Ability—Can the AI solution adequately perform the task it is designed for robustly, safely, and reliably?
Integrity—Will the AI solution properly process data provided to it, or is there a chance the information could be manipulated in a harmful way?
Benevolence—How does the AI solution adhere to “do no harm” principles?

One significant factor that affects AI solutions is its ability to access the internet, particularly in applications that are integrated with LLMs.

Some controls for this situation include:

Deny access—Depending on the use case, enterprises should not allow AI models to access the internet.
Implement access controls—Access controls are a key control for AI solutions. If an application needs to access the internet, organizational policies regarding internet access should be followed.
Block restricted websites—Restricted websites should be blocked from the application using tools such as URL or spam filtering.
Control access to available data—The application should be limited to what data it is allowed to access based on approved use cases.
Ensure data validation—The data used by the LLM should be reviewed to ensure it is accurate and not harmful.

3.13.6 AI Acceptable Use Policy

Creation of an AI acceptable use policy (AUP) is a required security control for the organization to state the framework for ethical and responsible deployment and use of AI. It should provide clear guidance on usage that balances its benefits against risk. The policy should include expectations for the use of AI tools. Similar to a general AUP, the AI AUP communicates required and prohibited activities and behaviors. It is the big picture of what and why that establishes the intent about a particular topic, reflecting the broader goals, objectives, and culture of the organization. It is a tool to inform staff of expectations: what is and is not allowed.¹⁸¹ See 1.10 AI Acceptable Use Policy for more information.

3.13.7 AI Audits and Traceability

Audits are a critical element for any technical implementation to ensure requirements and policies are being followed. Similar to access control, many organizations are using AI toolsets for automatic audits and traceability assurances. The human factor is still critical at this stage of an organization’s adoption of AI. The potential complexity of multiple layers of AI models, along with different groups using AI specific to their department, brings significant risk to the governance of AI.

Traceability in AI is the ability to track how data moves through the system and how it makes decisions. Traceability can be useful to verify the origin of data, processes, and other factors that are key to developing an AI model.¹⁸² Regular audits can help ensure traceability to avoid concerns around the black box nature of many AI models, especially proprietary models. Audits provide assurance, outside of the team that developed the model, that the algorithm and model are working as intended and suggest remediations for any findings related to lack of transparency or noncompliance.¹⁸³

Metadata logging is another way to ensure traceability in AI solutions. Metadata provides information needed about AI models, outputs, inputs, and other details that help to ensure the trustworthiness of AI solutions.¹⁸⁴ Model cards are one method used to record this information. See 2.4.2 Model Cards for more information. Software solutions are also available to help capture metadata related to AI models.

3.13.8 Shadow AI

Shadow AI, the use of unapproved AI applications, has serious consequences. Shadow AI introduces significant risk, including accidental data breaches, compliance violations, and reputational damage. Shadow AI may also lead to direct violations of regulatory requirements (e.g., EU AI Act), exposing organizations to compliance penalties if unapproved systems process sensitive data. These unapproved applications should be identified, assessed, and either removed or integrated into the existing security architecture.

Guardrails need to be created to ensure AI applications and tools can be identified to reduce or remove the risk of proprietary data leaking into public domain models. Once proprietary data gets into a public domain model, more significant challenges begin for any organization. Consider these controls to identify and protect against shadow AI:¹⁸⁵

A shadow IT policy
IT department as a service delivery organization
IT budgeting and procurement
IT system consolidation
User education
Cloud access security brokers (CASBs)
DLP
Network analysis tools
UEBA
Dataflow monitoring
Web filtering
URL blocking

3.13.9 Prompt Templates

Prompt templates, which are used to control data input into the AI model, standardize and sanitize instructions and limit the variety in which the model takes instructions. Prompt templates can be effective in preventing or reducing the risk of prompt injections.

Prompt templates can be used as an additional preprocessing step before the data input is fed into an AI model. The template also helps the model achieve higher output performance because of the standardization.

3.13.10 Adversarial Testing

Adversarial testing of AI is similar to security red team or ethical hacking activities. By intentionally feeding malicious data inputs into the model, testers using adversarial testing techniques can elicit unexpected and incorrect responses from an AI system to test it and the underlying model’s resilience to malicious threat actors or edge case scenarios.

For a more thorough knowledge base of adversarial tactics and techniques, the MITRE corporation has developed the Adversarial Threat Landscape for Artificial-Intelligence System (ATLAS) to capture a knowledge base that can be leveraged by AI red teams (figure 3.18).¹⁸⁶

See 2.7.2 Adversarial Training for more information.

A matrix-style diagram shows the MITRE ATLAS framework. — Figure 3.18—MITRE ATLAS

3.13.11 Defensive Distillation

Defensive distillation is a detection technique to guard against adversarial input to an AI model. This technique works by training two models—an original teacher model and a second distilled model, which is trained from the results of the teacher model and later used for inference. This allows the resulting distilled model to be more robust and resilient against an adversarial attack.

3.13.12 Regularization

Regularization prevents the overfitting of an AI model. While it is often used to improve the model performance by generalizing well, regularization also helps in defending against adversarial attacks. Some attacks use small changes to data to infer a model’s features and the tolerances within which a prediction output could change. To prevent this type of attack, the regularization technique makes the boundaries less defined and less susceptible to inferential types of attacks.

3.14 Use of AI in Control Management

AI can be a valuable tool for automating parts of the control management process. For control selection, AI can analyze historical data and provide suggestions on what types of controls are best suited to manage risk to acceptable levels.

AI automation takes over repetitive tasks like monitoring and scanning and can assist in analyzing data-heavy logs. Reduction in human error provides the security team with the space to focus on complex problem solving and strategic decision making.

For example, existing threat detection and response tools—such as SIEM; security orchestration, automation, and response (SOAR); and extended detection and response (XDR)—now leverage AI capabilities to enhance their functionality.¹⁸⁷ UEBA also employs AI to help with detecting anomalous user behavior and analyzing large amounts of user data related to networks and IT assets.