Part A: AI Risk Scenario Identification and Assessment

Identifying and assessing AI risk scenarios is a foundational step in managing the unique and evolving risk associated with AI systems. AI risk is inherently dynamic, shaped by rapid technological advancements, novel threat actors, and emerging vulnerabilities that differ significantly from traditional IT risk. The IT risk management life cycle remains relevant (figure 3.1) for addressing and managing risk associated with AI.

Risk identification remains an important first step for AI risk management. This starts with accurately inventorying AI use within the organization. See 2.15 AI Asset Inventory for more information. Once enterprises understand how AI is used, it can uncover risk associated with its use. AI-related threat and risk models can be leveraged to create risk scenarios that can be analyzed and used to inform risk treatment strategies for AI. This begins with understanding the AI threat landscape.

3.1 AI Threat Landscape

The AI threat landscape encompasses the risk, challenges, and security and privacy concerns associated with the development and use of AI solutions. The AI threat landscape is multifaceted and involves a wide range of threat actors attempting to exploit vulnerabilities in a new technology that provides exponential computing capabilities. Their goal is either to exploit the AI solution itself or leverage it to harm people, organizations, or the ecosystem.

To manage AI risk effectively, practitioners must understand the unique threat landscape introduced by AI technologies. These include both traditional cybersecurity threats and novel AI-specific threats such as:

• AI regulatory risk—Fragmented or inconsistent AI regulations across regions could constitute threats to organizations.
• Adversarial attacks—Specific attacks are designed to trick the AI system. For example, a self-driving car might be tricked into misinterpreting a stoplight as “go,” or a threat detection system might view legitimate traffic as hostile.
• Data poisoning—Adversaries manipulate the training data used by the model. This can lead to biased, inaccurate, or malicious outputs.
• Data privacy and security—AI systems often process sensitive data. Breaches or misuse of this data can lead to steep legal penalties, harm brands, cause reputational damage, lead to financial losses, and destroy trust.
• Insider threats—When it comes to AI systems, malicious or negligent insiders can cause significant damage by stealing sensitive data, model theft, sabotaging models, or disrupting system functions or operations.
• Lack of explainability/transparency—The inability to share details or taking a black box approach to AI models not only makes it challenging to identify the root cause of errors or biases but also erodes trust. This makes it difficult to demonstrate regulatory compliance and to ensure value alignment with ethical considerations.
• Model theft/reverse engineering—An enterprise’s model (the algorithms and parameters) can be stolen. This allows adversaries to replicate unique AI-driven services without the need to invest development resources.

See 3.1.1 Vulnerabilities and Attacks Specific to AI for more detail on each of these areas.

In addition to threats targeting the AI models themselves, organizations must be aware of systemic risk introduced through external dependencies and operational constraints, including:

• Supply chain vulnerabilities—AI systems rely on components from a variety of vendors (e.g., data sources for training models, graphics processing units [GPUs] and supporting hardware, software libraries). A vulnerability in any of these can lead to the entire AI system being compromised. This has become a very large category in contrast with traditional supply chain concerns. Enterprises must consider not only vulnerabilities in software but also potential hardware alterations or compromised datasets and pipelines. See Part E: AI Supply Chain Risk Management for more information.
• Vendor lock-in/dependency—Sole sourcing to a single AI provider creates vendor lock-in. Should that provider experience an incident (e.g., data breach, ransomware, outage), the organization could be impacted. See 3.17 AI Vendor Management for more information.

These threats correspond to risk categories that are familiar to enterprises:

• Competitive—If a competitor steals training data, prompt logs, or model parameters, or if it uses adversarial attacks to disrupt AI-powered services, the enterprise could lose stakeholder confidence as well as market share.
• Fines/Penalties—If enterprises are found to be operating AI solutions in a noncompliant manner, violating data privacy regulations or other AI-related laws, legal action can be taken and financial penalties imposed.
• Productivity—Systems are designed and deployed to support and enable business processes. System failures can lead to significant operational downtime.
• Reputational—Loss of stakeholder and public trust can result from operational failures or security incidents.
• Response—Costs associated with recovering from data breaches, service disruptions, or the cost of recovering from attacks can affect the long-term sustainability of an enterprise.

Figure 3.2 shows examples of threat actors and related AI threat scenarios.

3.1.1 Vulnerabilities and Attacks Specific to AI

Recent advancements in generative AI (GenAI) models have ushered in new use cases, modalities of interaction (e.g., text prompts, voice commands, visual signs), and new threat vectors. Such a large addressable market has attracted threat actors that are intent on exploiting the weaknesses that stem from rapidly adopting new technologies with immature guidance.

As AI systems are software applications at their core, OWASP has developed a top 10 list of security issues for large language model (LLM) applications.¹⁴⁰ Many attacks are similar to those that organizations commonly see with IT or cybersecurity programs. How OWASP ranks common LLM risk is shown in figure 3.3.

Besides the general AI risk that should be addressed through governance, according to the OWASP AI Exchange, AI threats fall into three major attack surfaces:¹⁴¹

1. Development-time threats
2. Runtime security threats
3. Threats through use

Development-time threats (figure 3.4) occur during the development of the AI system. Attack surfaces include the engineering environment and the supply chain.¹⁴²

Runtime application security threats are attacks that occur due to security weaknesses that are not AI-specific. AI systems are also IT systems, so common IT vulnerabilities associated with software wrappers or other IT infrastructure used to host an AI solution should be considered.¹⁴³

AI threats through use occur during normal activities associated with the use of an AI model, particularly input and output activities.¹⁴⁴

Training Data Leakage

During the development of an AI solution, data can be leaked or stolen from the training environment or data store. This can occur because of weak access controls in one or more systems that are in the data supply chain for the development of the model. It can occur at the originating data source, at the platform where data cleansing or preprocessing is performed, or at individual development environments, like Jupyter Notebooks.

While data access is limited to authorized data scientists and engineers, the training platform may not have implemented data exfiltration controls to prevent users from accidentally or maliciously copying data outside of the authorized training platform.

For example, popular interactive coding and computing platforms, such as Jupyter Notebooks and Google Colab, allow data scientists to document and run Python code to explore, process, and perform other data science tasks. Real data may be pulled into these environments, and the code and outputs (with real data) can be saved locally or in cloud computing environments, creating data sprawl and allowing data leakage. It is difficult to prevent and detect such scenarios, because a simple line of Python code can easily exfiltrate data outside of the organization’s control. Robust controls, including infrastructure and networking access controls, paired with application-level monitoring, need to be designed and implemented to address this new threat vector.

Data Poisoning

Malicious or erroneous data inserted into datasets with the intent of changing an AI model’s output behavior is called data poisoning. For large datasets, it is challenging to discern poisoned data from good data. There are multiple entry points where poisoned data can be introduced into the training dataset, including:

• Source—Data can be changed at the source where training data is collected.
• Supplier—Some organizations acquire training data from a data supplier or data broker, and data can be changed on the supplier’s end or during transfer.
• In transit—AI data pipelines traverse multiple intermediary systems, and attackers can breach those systems to poison the data.
• Preprocessing—Data preprocessing and cleansing routines can be modified to poison the data.
• In-context training—An external knowledge base used in RAG grounding techniques to augment LLM data inputs can be poisoned.

Data poisoning can lead to two key categories of risk impact: 1) creation of a backdoor or Trojan horse to modify a model’s specific behavior or output or 2) sabotage of the model to undermine safety, trust, or performance.

Model Poisoning

There are three potential methods for poisoning an AI model: 1) poisoning the data; 2) directly changing the model’s parameters, architecture, or libraries used for training; and 3) poisoning the model from a model supplier, if an organization uses a third-party, pretrained model for further training or fine tuning.

In order to change a model’s parameters, architecture, or libraries, an attacker would need to access the development or production environments to manipulate the source code, configurations, or other components used to train the model. Access controls should be put in place and reviewed regularly to prevent this type of attack.

Model Theft

The development of an AI model is a significant investment for many organizations, sometimes costing millions of dollars; therefore, enterprises aim to protect this form of intellectual property (IP) or proprietary information. Stolen AI models can be used to learn how the models behave to develop further adversarial attacks against them.

At its simplest form, an AI model is a mathematical algorithm that has been designed with a certain architecture (e.g., transformers, generative adversarial networks [GANs], recurrent neural networks) and weights of the features derived from data. The model is represented as source code files, configurations, and binaries. To steal a model, an attacker would need to gain access to the system (or code repository) in which the files and binaries are stored or loaded into memory, usually through query-based attacks.

An AI model can also be stolen during runtime or through use if the attacker does not have direct access to the model source code, binaries, and configuration file. By sending specifically crafted input to the model under attack and analyzing the resulting output, the attacker can infer the model’s parameters, features, and even the training data used. The attacker is then able to reconstruct a copy of the model.

Prompt Injections

A prompt injection is an adversarial attack against a GenAI model. This type of attack uses specifically crafted text prompts that provide specific instructions to the AI model. It is often designed to get the model to ignore certain aspects of the original instructions and follow the attacker’s instructions instead. It is similar to the classic Structured Query Language (SQL) injection attack on databases. Prompt injections are popular because they are easy to execute and do not require additional privileges to access the system or data. They instead use the available AI solution user interface or application programming interface (API) calls that are normally provided. An indirect prompt injection can also be executed when the model ingests malicious instructions from another data source (e.g., ground into the prompt via RAG).

Prompt injections can be mitigated by leveraging prompt templates that force structure and allow for sanitization of the original prompt. Prompts that fall outside of the defined parameters are rejected or adjusted before they are fed into the model for inference. Similarly, prompt injections can be mitigated during runtime processing when unapproved or inappropriate outputs are prohibited despite the prompt instructions.

Model Evasion

Model evasion occurs when an attacker modifies the data specifically to circumvent the correct prediction of an AI model and cause errors. This leads to misclassification, misdetection, or some other unintended behavior. For example, an attacker learns the features and tolerances in an automated email protection system and modifies a malicious email’s contents to be erroneously perceived as safe by the system.¹⁴⁵ The malicious email is then misclassified by the AI model as “safe” because the email contents contain key elements that result in a high safety rating.

Model Inversion

Whereas model evasion is intended to fool the model, model inversion occurs when an attacker tries to extract sensitive or private information from the model. The attacker studies the model’s data input and resulting output, makes tweaks to the data input, observes the model’s response to those changes, and infers the model’s inner workings. The attacker can train a separate AI model that takes the output of the AI model under attack and deduces what the original input data was. Through numerous attempts and systematic optimization, the attacker can invert the model’s function.

Through model inversion, the attacker can infer sensitive attributes about the data used in the training of the model under attack (attribute inference), infer whether a specific data record is contained in the training dataset (membership inference), or reconstruct a high-quality copy of the training dataset.¹⁴⁶ Mitigation approaches include applying differential privacy techniques to protect sensitive data used in training.

Threats for Using Vendor-supplied AI

Many organizations may not have the resources (e.g., technical talent, computing resources, finances) to train or fine tune their own AI models. Vendors can bear the burden of training and supplying their AI models and solutions in the hope of recuperating those costs and making a profit through sales to customers. Those vendors also bear the responsibility of securing their AI models and solutions from AI-related threats.

The OWASP threat model (figure 3.5) adapts the AI security threats and controls for scenarios where an organization is using a GenAI model out of the box (OOTB).

AI Solution Disruption

As AI becomes more pervasive in daily life, enterprises and individuals are becoming more dependent on the availability of AI solutions. An attacker can wreak havoc by sending either a large volume of inference requests for the AI solution to process or by sending computationally intensive inference requests. These tactics can result in a denial of service (DoS) or degradation of the AI solution.

3.1.2 Nontechnical Threats

The accelerated adoption of AI presents new layers of threats that cannot easily be addressed using a traditional technical approach. Ethical concerns cannot be patched, and there is no hot fix for biased decision making. Also, these areas may not be covered by existing risk assessment procedures. Therefore, enterprises must be aware of nontechnical AI threats (figure 3.6) so they can be included in the risk management process.

3.2 AI Threat Modeling

Threat modeling is an important part of the risk analysis process, as it is important to identify the variety of actors associated with a given scenario (not necessarily with the same level of rigor as attribution, although positive attribution can accelerate the process).

It is important to understand which controls are successful at defending against the attacks or errors associated with various threat communities. In some cases, controls may address threats from more than one source. For example, the same control that addresses nation-state-sponsored attacks may also defend against cybercriminals. Overlapping controls in this sense should be noted in the threat model.

In general, threat modeling will:

• Create an abstraction of the system
• Profile attackers, including goals, methods, and attacker skills
• Create a catalog of potential threats

Additionally, it will provide high-level answers to basic questions, such as:

• How often will a threat actor encounter an asset?
• What is the danger of discovery posed to that threat actor while it is attempting to compromise those assets?
• What is the perceived value for the adversary?
• What skills does an adversary need to succeed in compromising systems or gaining access to data?
• How much time does an adversary need to compromise systems and access data?
• What resources and materials would the adversary need?
• What level of effort is required overall from the threat actor?

Several threat modeling methods can be combined to create a more robust and well-rounded view of potential threats. Not all methods are comprehensive; some are abstract, and others are people centric. Some methods focus specifically on risk or privacy concerns. These methods can also be leveraged by AI agents to automate some components of the threat modeling process with human oversight. Figure 3.7 provides a list of threat modeling methods.

As noted above, it is important to adapt existing threat modeling techniques to include considerations for ethics, privacy, and human rights, which are not commonly included in traditional IT risk and threat assessment.

As with many areas of IT and cybersecurity, use cases are evolving for the use of AI to assist with many risk management tasks, including threat modeling. Tools, such as STRIDE GPT and Irius Risk, leverage AI to help teams create risk and threat scenarios based off common frameworks.

3.2.1 Using AI for Threat Modeling

The use of AI in threat modeling represents a significant advancement in automating and augmenting traditional risk identification and assessment activities. AI technologies can enhance the efficiency, accuracy, and adaptability of threat modeling by generating detailed threat profiles, simulating complex attack paths, and identifying emerging threats in dynamic environments.

Automating Threat Profile Generation

AI can assist in creating and maintaining comprehensive threat profiles by analyzing vast amounts of data related to threat actors, their motivations, methods, and skills. By leveraging natural language processing (NLP) and machine learning (ML) techniques, AI tools can extract relevant information from diverse sources such as threat intelligence feeds, security reports, and incident logs. This automation supports the continuous updating of threat profiles, ensuring that organizations maintain an up-to-date understanding of the threat landscape relevant to their AI systems and broader enterprise assets.

Simulating Attack Paths and Scenarios

AI-driven simulation models enable the exploration of potential attack paths by mapping adversarial tactics, techniques, and procedures (TTPs) against system architectures and vulnerabilities. These simulations can incorporate AI-specific threat considerations, such as adversarial ML attacks, data poisoning, and prompt injection attacks, which traditional threat modeling methods may not fully address. By integrating AI capabilities, organizations can generate more realistic and complex threat scenarios that reflect the evolving nature of AI-enabled attacks.

Identifying Emerging and Adaptive Threats

The dynamic and rapidly evolving AI threat landscape requires continuous and adaptive threat modeling. AI-powered tools can analyze patterns and anomalies in security telemetry, user behavior, and external threat intelligence to detect novel or emerging threats that may not be captured by static threat models. This proactive identification supports timely risk mitigation and informs strategic decision making.

Benefits and Considerations

Using AI for threat modeling offers several benefits, including scalability to large and complex systems, improved accuracy through data-driven insights, and the ability to adapt to new threat vectors rapidly. However, organizations must ensure that AI-assisted threat modeling incorporates ethical, privacy, and human rights considerations, which are often underrepresented in traditional models.

Human oversight remains essential to validate AI-generated threat models, interpret results in context, and make informed risk management decisions. The combination of AI automation and expert judgment creates a robust approach to managing AI-related risk effectively.

3.3 Development of AI Risk Scenarios

Constructing detailed AI risk scenarios is a critical step in understanding and managing the complex and evolving risk associated with AI systems. An AI risk scenario serves as a practical tool for communicating risk effectively to business process owners and stakeholders, enabling informed decision making aligned with organizational goals.

An AI risk scenario includes:

• Threat event—A specific security incident or adverse event related to AI, such as unauthorized data disclosure, model manipulation, or operational disruption
• Threat actor—The internal or external party responsible for the threat, which includes insiders, cybercriminals, nation states, or AI developers
• Affected assets—The people, organizational structures, IT and physical infrastructure, data, applications, or processes impacted by the threat event
• Potential impact—The primary and secondary losses that could result, including financial loss, reputational damage, regulatory penalties, or operational downtime
• Frequency and likelihood—An assessment of how often the threat actor and asset interact, the probability of a successful attack, and the likelihood of loss realization

To be effective, AI risk scenarios must be relevant and realistic, reflecting the organization’s specific context and risk appetite. Scenarios should be directly linked to the enterprise’s objectives and priorities, ensuring that risk management efforts focus on events that could materially affect business outcomes. This alignment helps avoid an overwhelming number of hypothetical “what if” scenarios and instead concentrates resources on high-impact, plausible risk.

Developing AI risk scenarios benefits from a balanced approach that leverages both historical incidents and emerging AI threats. Historical events provide a foundation for understanding known vulnerabilities and attack patterns, enabling organizations to prevent recurrence and improve resilience. However, exclusive reliance on past events is insufficient given the rapid evolution of AI technologies and threat landscapes. Scenarios must also anticipate novel risk, such as those arising from GenAI misuse, prompt injection attacks, or systemic AI failures, to prepare the organization for future challenges.

3.3.1 Risk Scenario Development Process

The process of developing AI risk scenarios typically involves:

• Collaboration—Engaging diverse stakeholders—including senior management, business units, IT, risk management, and subject matter experts—to gather insights and achieve organizational buy-in
• Creativity and analysis—Employing brainstorming and structured workshops to explore potential threat events, their causes, and cascading effects
• Scoping—Maintaining a manageable number of scenarios by focusing on those with significant potential impact and relevance to critical business units and processes
• Review and update—Regularly revisiting scenarios to incorporate changes in technology, threat actor capabilities, regulatory requirements, and organizational priorities

3.3.2 Using AI for Risk Scenario Development

AI, GenAI in particular, can be leveraged to aid in the development of risk scenarios for risk management. GenAI and LLMs can process large amounts of data in a short amount of time, making them useful for scanning threat and risk reports and scraping news sites for details regarding emerging threats.¹⁴⁷ AI can also enable risk management teams to focus more on decision making, as the data analysis piece of risk scenario creation will be automated. These insights can be leveraged to create more robust risk scenarios. AI-enabled risk scenario development has been found to be of particular benefit for regulatory and compliance management.¹⁴⁸

For example, a study was conducted where a multiagent system was used to search over 100,000 news sources using a Boolean query created by an LLM.¹⁴⁹ The result of the search was 39 relevant news articles. Then, LLM prompts were used to chunk and analyze the data in the articles to define an “influence event”—a summary of the title, description, and actors in the article that may be relevant to the search query. This data was further classified based on relevance. After this review, nine events were identified and then further processed to create 37 projections based on these events. The projections were fed into the LLM again in pairs, analyzed, and used to create risk scenarios.¹⁵⁰

The limitations of using AI for risk scenario generation are similar to any use of AI solutions (e.g., hallucinations). They include bias in training data scraped from external and news sources, data privacy concerns, and technical/skill limitations, such as creating prompts to derive a Boolean query (as in the example).

3.4 AI Risk Classification

As with other technology risk, the enterprise will need to establish categories of AI risk based on its objectives and strategy. The enterprise will also need to review what level of risk it is willing to accept related to the use of AI, as it may differ from existing risk tolerance and appetite levels.

Various factors can inform these decisions.

3.4.1 Risk Classification

Risk classification categories continue to evolve and mature over time as advancements are made in the field of AI and risk. Enterprises can leverage existing risk taxonomies and evaluate if they are robust enough to cover risk related to AI. Risk management can also look to the growing body of AI risk taxonomies and classification categories suggested by academic, institutional, and governmental bodies.

The MIT AI Risk Repository

The MIT AI Risk Repository is a living database of over 1,600 AI risk areas, which have been categorized based on causal and domain-level taxonomies.¹⁵¹ The purpose of the repository is to provide a common vocabulary and frame of reference for discussing AI risk.

The causal taxonomy of AI risk is shown in figure 3.8.

The seven domains covered in the domain taxonomy of AI risk are shown in figure 3.9.

The EU AI Act

The EU AI Act identifies risk categories to provide organizations with an easy way to understand what kind of risk is permissible and what must be managed more rigorously:¹⁵²

• Unacceptable risk—AI applications that manipulate behavior, violate rights, or engage in social scoring are prohibited. These are AI applications that could cause severe harm, such as those that manipulate human behavior in ways that are unethical. Such systems are not allowed at all.
• High risk—AI systems that require strict oversight. These include AI solutions that can significantly affect people’s lives (e.g., those used in healthcare, law enforcement, critical infrastructure). These require strict controls, extensive documentation, and regular assessments, testing, and reviews to ensure they do not overstep safe boundaries.
• Limited risk—AI systems that cannot materially harm an individual. These are systems like chatbots, which might influence decisions but are less likely to cause major harm. However, enterprises need to be transparent in their use of these types of AI. Their tasks are of such a narrow and limited nature that they pose only limited risk, which is not increased through the use of an AI system. For example, a chatbot used for customer service may pose limited risk due to its predefined responses, whereas a predictive AI system used in financial decision making may introduce broader risk related to bias, compliance, and accountability.
• Low/No risk—These are AI systems with little or no regulatory oversight required, such as spam filters.

This simplistic, qualitative view of risk categories may provide a suitable classification system for enterprises that are less dependent on the financial or quantitative impacts of AI.

FAIR-AIR

The Factor Analysis of Information Risk-Artificial Intelligence Risk (FAIR-AIR) Approach Playbook leverages the FAIR risk management framework and translates AI-related risk into financial terms. This enables effective communication about risk to senior leadership and other relevant stakeholders.¹⁵³ Compliance-driven enterprises can map FAIR-AIR outputs to jurisdictional regulatory frameworks like the EU AI Act.

Figure 3.10 shows the five steps in the FAIR-AIR approach to risk classification.

The FAIR-AIR approach allows organizations to classify and determine if AI-related risk is within acceptable limits or if it exceeds defined thresholds. These standard risk classification processes include:

• Contextualize—First, understanding the driver behind risk analysis ensures that the enterprise is clear about its business goals for AI. It is important to understand who the assessment is for, what decisions are under consideration, and why.
• Scope—Next, the most relevant area of the AI solution is examined. The enterprise identifies what needs to be protected (the resource/asset) and then studies the potential threats and their impacts. Once identified, specific risk scenarios can be analyzed based on what matters most to the identified stakeholders.
• Quantify—Once risk scenarios are identified, various methods can be used to gather sufficient data (both internal and external). The FAIR approach is useful in calculating how much risk (or potential loss) exists for each scenario developed. Quantification provides clear and concise information on the scenario’s risk exposure in financial terms.
• Prioritize/Treat—Once risk exposure is determined, risk management can collaborate with the appropriate stakeholders to rank each scenario based on the likelihood of each risk occurring and the resulting financial impact if the risk were to be realized. This is often used to identify the driving factors for each risk scenario when deciding on effective risk treatment options. This helps determine which risk the organization should address first and how to allocate its limited resources.
• Decision making—Stakeholders can then decide how to make investments that will have the largest impact and provide the best value to the enterprise.

3.4.2 Risk Appetite and Tolerance Levels

An organization’s owners and/or governing body are ultimately responsible for establishing its risk appetite and tolerance criteria. Defining acceptable limits requires a balancing act that considers multiple influences. Figure 3.11 shows some of the factors that should be considered when establishing acceptable limits.

Each organization needs to consider what risk it is willing to accept and identify compliance requirements (e.g., EU AI Act, the Health Insurance Portability and Accountability Act [HIPAA], General Data Protection Regulation [GDPR], etc.) that may compel them to demonstrate conformance with a set of obligations.

Acceptable levels of risk are established as part of an organization’s defined information security strategy, with management determining acceptable risk levels, normally defined by the losses the organization is willing to accept in pursuit of its goals and objectives. Other rationales for acceptable levels of risk include enterprise risk appetite and tolerance levels.

3.5 AI Risk Assessment

In many ways, AI risk is the same as all other risk the enterprise faces. Therefore, AI risk should be integrated into existing risk assessment processes. Risk assessments should be performed on a regular basis and during any changes to AI solutions. Leveraging the organization’s current definitions for risk can be a good starting point to assess the likelihood and impact of identified risk to the organization. Special considerations for the unique attributes of AI solutions should be documented within the process. If these attributes are not present in current risk assessment processes, the organization should update the processes accordingly.

Specific risk assessment considerations for AI solutions include:¹⁵⁴

• Methods and metrics—Clearly document metrics for measuring AI risk and trustworthiness characteristics identified during the risk identification phase. Ensure that independent experts are consulted to assess the conclusions of frontline developers.
• Evaluation of trustworthy characteristics—Evaluate the trustworthy characteristics of AI solutions (e.g., ethical use, bias and fairness, transparency and explainability, trust and safety, data security considerations [IP and personal data], human rights, and environmental impact).
• AI risk tracking mechanisms—Proactively identify approaches to track existing, unanticipated, and emerging AI risk.
• Feedback about efficacy of measurements—Ensure that facilitation of stakeholder feedback is considered in evaluation metrics.
• Operational risk—Evaluate how the AI system performs in real-world conditions and how it will be maintained.

3.5.1 Risk Assessment Methodologies

Risk assessment for AI solutions follows many of the same approaches as other IT implementations. However, the use of AI presents a level of nontechnical risk considerations that may be new for many enterprises, such as considerations of ethics, bias, fairness, and human rights (see Part F: AI Trustworthiness, Ethical, and Societal Implications for more information). In some cases, it is best to use a hybrid risk assessment that combines quantitative financial loss estimates and qualitative evaluation of ethical/social AI risk. Additionally, the large amount of data used to train and operate AI solutions raises numerous privacy concerns.

These emerging areas of risk may seem difficult to quantify and analyze, but many traditional risk assessment methodologies can be applied to these areas to help enterprises assess and prioritize AI-related risk.

Quantitative Risk Assessment for AI

Quantitative risk assessments tend to be the most useful in risk analysis, as they provide stakeholders with concrete loss information, typically in the form of financial impacts. For AI, existing procedures should be identified to determine the financial impacts of AI system failures.¹⁵⁵

Impacts of data loss and leakage as well as traditional IT and cybersecurity concerns, such as system downtime and loss of productivity, can be quantified for AI risk scenarios. Increased data storage needs and utility demands can also be measured.

Qualitative Risk Assessment for AI

As noted, AI presents unique areas for consideration, namely in the ethical, societal, and environmental impacts its use can have. These areas may also be more difficult to quantify and may have more significant impact on an enterprise’s reputation and trustworthiness. Segmented global, regional, and industry standards and regulations mean that the risk of an ethical incident can have varied impact depending on where it occurs.

3.5.2 Fundamental Rights Impact Assessment for High-risk Systems

Faults in AI solutions, such as bias and discrimination, can potentially harm humans. Most commonly, this harm relates to privacy, data protection, and equality. Infringement of human rights can lead to reputational damage for AI users (e.g., companies, individuals) and potential litigation and related financial risk.

Before deploying any high-risk AI system (one that poses a significant threat to the health, safety, or fundamental rights of individuals), especially one used by public bodies or private entities providing public services, the EU AI Act requires a fundamental rights impact assessment (FRIA). This evaluation is designed to ensure that the system does not negatively affect an individual’s rights.

At a high level, the assessment process involves:¹⁵⁶

• Usage description—Context of how the AI system will be used
• Operational details—When and how often the AI system will be used
• Affected groups—Who may be affected by the use/operation of the AI system
• Risk identification—Specific risk that could affect individuals
• Oversight measures—How oversight will be maintained (human in the loop [HITL])
• Mitigation plans—Procedures to address/respond/treat identified risk

The assessment must be completed before the AI system is first used and must be updated if any conditions change. The findings must be submitted to the appropriate regulatory authority.

The goal of the assessment process is not only to safeguard and protect the fundamental rights of humans affected by the use of AI but also to ensure transparency, accountability, and fairness in AI solutions. While similar to a FRIA, a data privacy impact assessment (DPIA), as required by the GDPR, does not fully cover the scope of the rights mandated by a FRIA, as FRIAs are broader in scope and should be conducted in addition to DPIAs where both apply.

3.5.3 Conformity Assessments

Conformity assessments demonstrate how AI systems are compliant with regulations, specifically the EU AI Act. These assessments evaluate both a provider’s quality management system and technical documentation. Under Article 43 of the EU AI Act, evidence that an assessment was performed is required for all high-risk AI systems,¹⁵⁷ with new assessments required after any substantial modifications.

While mandated by the EU AI Act, this assessment is useful for any enterprise implementing an AI solution. A conformity assessment can be thought of as a safety check for high-risk AI systems. It helps to ensure that AI systems are built and operate in accordance with the defined requirements and rules needed to protect an individual citizen’s safety, rights, and overall well-being.

Based on the type of system and the standards used, this assessment can be performed by the provider’s organization or through the external review by an independent expert. The system’s design, performance, and management processes need to be documented and carefully examined. The goal of this process is to build trust, ensure the system is reliable, and minimize risk before the AI system is used.

The conformity assessment ensures that an AI system meets regulatory and technical standards before and after deployment. Its processes are focused on the provider organization’s quality management system, technical documentation, and ongoing surveillance. Its goals are to mitigate risk, enhance transparency, and build trust in AI systems.

3.5.4 Privacy Impact Assessment

A privacy impact assessment (PIA) is an analysis of how personal information is collected, used, shared, and maintained for a specified scope of consideration. PIAs, referred to as DPIAs under GDPR, are structured processes for identifying and mitigating privacy risk, including risk to confidentiality. Their scope may span physical processes, administrative processes, technical processes, or a combination of these. In AI systems, specific risk such as inference attacks, reidentification through model outputs, and cross-dataset linkage must be considered when conducting PIAs.

At a high level, a PIA has four general goals:

1. Ensure conformance with applicable legal, regulatory, contractual, and policy requirements for privacy.
2. Determine threats, vulnerabilities, effects (harms and problems), and resulting risk.
3. Evaluate protections and alternative processes for mitigating identified privacy risk.
4. Develop a prioritization process for implementing the mitigation processes and associated privacy practices.

A PIA should be performed using a methodology that is established through a government or industry standard or regulation. Using a methodology ensures that the PIA process is repeatable and can be consistently performed from one PIA to the next.

New PIAs should be conducted whenever there is a change that is likely to involve a new use for personal data or significantly change the way it is handled, such as through implementation of new types of technologies, including AI solutions.

While traditional privacy risk focuses on unauthorized access, data breaches, and regulatory noncompliance, AI introduces additional concerns. AI-driven privacy risk includes algorithmic bias (unintentional discrimination against certain groups), inference-based privacy risk (inference of sensitive personal details beyond those explicitly provided), and lack of explainability in AI-driven decision making, all of which make it difficult to audit privacy compliance. These factors must be considered when conducting PIAs for AI systems.

AI systems often use data from other locations within or external to the team developing and using the solution. It is important to ensure that data protection principles applied to data at its original location persist to its new location in the AI solution.

3.5.5 Using AI for Risk Assessment

Since a key part of risk assessment is analyzing large amounts of data, AI can be a powerful tool to assist in the process. LLMs can be used to review interview data, summarize key risk considerations, and provide quantitative analysis of likelihood and impact.¹⁵⁸

Benefits of using AI for risk assessment are similar to the benefits seen in other AI use cases. These include automated monitoring and reporting, increased efficiency, the ability to identify and predict risk in emerging areas, and cost savings, among other areas.¹⁵⁹ Likewise, limitations include ensuring the use of high-quality, unbiased data in predictions and common AI-related concerns, such as fairness and transparency, skills of existing employees to use AI, and regulatory and compliance concerns.

Best practices for using AI for risk assessment include:¹⁶⁰

• Using well-defined use cases, such as interview summaries
• Ensuring human oversight where appropriate and where decisions require human judgment
• Developing a library of prompts for consistency
• Including feedback from other departments to enhance risk scenarios