Part E: AI Supply Chain Risk Management

From utility services and shared infrastructure to service providers and business partners, no organization operates autonomously. Enterprises rely on a wide range of third parties to operate. Within the course of normal operations, these relationships introduce unique risk (figure 3.23).

Widespread adoption of AI has led to organizations creating relationships with new vendors, but it can also impact existing relationships. It is important to understand the nuances of AI’s effects on these relationships.

First, it is critical to understand the threats and shared responsibilities typical of vendor and supplier relationships. Understanding the context of an AI system’s use within its operating environment is important, as context can have different implications for an organization’s response strategies and may shift legal liability and responsibility from one entity to another. Figure 3.24 illustrates this relationship.

3.17 AI Vendor Management

Existing vendor management programs should be reviewed and updated to include AI considerations, especially as many existing vendors may include AI features in their services with or without the prior knowledge of the enterprise. As a result, new risk can be introduced, including:

• Business risk associated with new AI features—Vendors may introduce new features and functions to increase productivity and gain efficiencies without considering the requirements of their contracting organizations. Managers should establish processes that require vendors to provide advanced notification in order to evaluate the potential of features to disrupt operations or require significant overhaul of existing workflows.
• Privacy risk associated with new AI features—Understanding how vendor models are trained is necessary to prevent introducing or exposing organizations to violations of privacy regulations.
• Security risk associated with new AI features—New features can lead to new exposures and vulnerabilities. Managers should gain sufficient assurances and guarantees that vendor security practices sufficiently test AI system changes to avoid introducing preventable vulnerabilities.
• Compliance with emerging regulations—Understanding how vendors adapt to changing regulatory requirements is necessary to avoid penalties. Managers should establish protocols to include general counsel, compliance, privacy, and security teams in order to understand impacts across differing jurisdictions.
• Concentration risk—Organizations should also assess concentration risk, where heavy reliance on a limited number of AI vendors creates systemic exposure if a provider fails.

Integrating AI-centric features into a vendor management program demonstrates a proactive, efficient, and comprehensive approach to how an organization manages AI-related risk. Integration is essential not only for protecting data but also for ensuring compliance, maintaining operational resilience, and fostering consumer and stakeholder trust.

Figure 3.25 illustrates a potential approach to AI vendor management. The steps in the example provide a means to potentially reduce organizational risk during the vendor vetting process, before the enterprise becomes so reliant on the vendor or contracted service that its discontinuation would disrupt or greatly impact the business.

3.17.1 AI Vendor Considerations

The following are common contributors and influencing factors for not selecting a vendor and should be considered during the normal vetting process:

• Financial instability
• Regulatory noncompliance
• Poor security practices
• Reputational risk
• Inadequate operational resilience

These factors can be applied to general technologies but may be amplified in the context of AI.

When evaluating the use of a third-party AI solution, primary concerns include:

• Lack of transparency and explainability in the AI model
• Unacceptable ethical risk
• High potential for data privacy violations
• Vendor lock-in
• Immature or unproven AI technology that impacts health or safety
• Lack of a risk management framework that incorporates AI
• Inability to meet AI security requirements
• Lack of content provenance mechanisms for GenAI
• Resistance to due diligence, auditing, and security assessments

There are some situations in which a provider or vendor is considered high risk, but an organization chooses it anyway because the associated risk can be adequately mitigated, transferred, or accepted.

On the other hand, an AI vendor could be found to exceed an organization’s acceptable risk appetite and tolerance criteria. Its inherent characteristics, lack of maturity, unwillingness to cooperate on risk assessment, or the nature of its AI technology’s use in the organization’s applications might create significant potential for negative impacts that the organization could not adequately understand, control, or mitigate through contractual terms, technical safeguards, or operational procedures. The expected benefits of using the AI vendor would then likely not outweigh the potential risk.

When selecting a vendor to deliver on AI services, several business considerations need to be carefully weighed to ensure a successful and beneficial outcome. These considerations span strategic alignment, risk management, data handling, technical capabilities, the vendor’s team and expertise, cost and ROI, the nature of the vendor relationship, and legal and ethical implications (figure 3.26).

By carefully considering and addressing these factors when considering and selecting an AI provider, organizations can make the most well-informed decisions possible, maximizing potential benefits while also keeping risk within appropriate boundaries.

3.17.2 Contractual Considerations

Figure 3.27 lists common elements of an AI vendor contract.

Vendor Lock-in

Vendor lock-in is a significant challenge and risk in AI supply chains, particularly when leveraging cloud-hosted AI solutions or proprietary AI platforms. Lock-in occurs when an organization becomes dependent on a single vendor’s technology, services, or infrastructure, making it difficult, costly, or operationally disruptive to switch providers or migrate data and workloads. This dependency can limit flexibility, increase costs over time, and expose the organization to strategic risk if the vendor’s service quality declines, pricing changes unfavorably, or the vendor discontinues support. Vendor lock-in also complicates compliance efforts, especially when data residency or jurisdictional requirements must be met, as migrating data across providers or regions may be restricted or technically challenging.

Operational impacts of vendor lock-in include reduced control over infrastructure and potential performance issues due to network latency or shared resource models inherent in cloud environments. Strategic impacts encompass diminished negotiating power, constrained innovation, and increased exposure to vendor-specific vulnerabilities or failures. Organizations should weigh this risk carefully when deciding between internal hosting and cloud-based AI solutions, considering factors such as scalability, cost, control, and alignment with long-term business objectives.

To mitigate vendor lock-in, organizations should:

• Negotiate service level agreements (SLAs) that include clear exit and data portability clauses.
• Maintain comprehensive documentation of AI system configurations and data schemas.
• Evaluate vendors’ interoperability and support for open standards.
• Plan for phased changeovers or hybrid deployment models to reduce dependency.
• Conduct ongoing vendor performance monitoring and risk assessments.

Open-source Software

The use of open-source software by enterprises and vendors alike creates additional considerations in AI.

It offers advantages such as accelerated development, cost efficiency, and access to a broad community of contributors. However, its use introduces unique challenges related to licensing compliance, security vulnerabilities, and the complexity of maintaining a comprehensive inventory and monitoring process.

Open-source components are governed by a variety of licenses, each with specific terms and conditions that dictate how the software can be used, modified, and redistributed. Failure to comply with these licensing requirements can expose an organization to legal liabilities, including IP infringement claims and potential financial penalties. In the context of AI, where models and codebases often incorporate multiple open-source software libraries, it is critical to conduct thorough license reviews and ensure that all open-source usage aligns with organizational policies and contractual obligations. Clear documentation and contractual provisions should address ownership and usage rights, especially when AI-generated content or code is involved, to prevent ambiguity in IP ownership. Legal counsel should be engaged to review open-source licensing implications during vendor selection and contract negotiation phases.

Open-source components can introduce security vulnerabilities into AI systems, as they are often widely used and publicly accessible, making them attractive targets for attackers. Vulnerabilities may arise from outdated libraries, unpatched bugs, or malicious code contributions. Given the layered nature of AI supply chains, where third-party and fourth-party vendors may also incorporate open-source software, the risk of supply chain compromise is amplified. Organizations should implement proactive vulnerability management practices, including maintaining an up-to-date inventory of all open-source components, regularly scanning for known vulnerabilities using reputable databases (e.g., National Vulnerability Database, Common Vulnerabilities and Exposures), and applying timely patches or updates. Continuous monitoring and integration of security controls across the AI supply chain are essential to detect and remediate risk before it impacts the AI system’s integrity or availability.

3.18 AI Shared Responsibility Model

When acquiring AI capabilities from external vendors, distinctions between providers and deployers become crucial for understanding responsibilities and managing risk effectively. Similar to common cloud’s shared responsibility model, an organization that leverages AI services from a provider should clearly understand and delineate the responsibilities shared between the provider and the organization (figure 3.28).

Defining and requiring a shared approach between entities will need to be addressed contractually with clearly defined expectations related to SLAs, incident response activities, and incident notification requirements.

3.18.1 AI Deployer Responsibilities

AI deployers are the consumers of an AI system. Their responsibilities include:

• Audits and testing—Regularly audit AI systems to review privacy and security control configurations and conduct penetration testing to identify gaps or weaknesses with deployed controls.
• Data governance—Implement strong data governance practices to ensure data quality, integrity, privacy, and security. This includes data provenance tracking to understand data sources and transformations.
• Identity and access management (IAM)—Implement strong access control and identity management systems to restrict access to AI systems and data.
• Incident response planning—Develop an incident response plan to address AI-related security incidents; this involves plan development, staff training, and testing of an AI-centric incident response plan.
• Model validation—Conduct rigorous testing and validation of AI models, including adversarial testing, to identify and remove bias and vulnerabilities.
• Supply chain risk management—Assess the security posture of the provider and its vendors, and thoroughly vet their security practices, financial stability, and reputation. Include security requirements, breach and incident notification, and privacy provisions in contracts with AI providers.
• Transparency and explainability—Ensure that users are fully aware of any interactions with an AI system. Choose models that offer some degree of explainability in their decision making and chain of thought; monitor performance and outputs for anomalies.

3.18.2 AI Provider Responsibilities

Providers are the entities responsible for delivering AI services to subscribers. They can be organizations that offer AI as a service (AIaaS) or entities that deploy AI systems. Their responsibilities include:

• Compliance and audit certifications—Adhere to relevant security and privacy standards.
• Data security—Protect training data, model weights, parameters, system logs, and inference-time artifacts from unauthorized access or modification.
• Incident response—Have a robust incident response plan in place that can respond to operational, security, and privacy incidents and coordinate effectively with subscribers.
• Model development—Adopt an ethics framework, follow secure coding practices to develop models that are resistant to attacks, and strive to use biased-reduced training data.
• Transparency and explainability—Provide AI notifications, tools, and supporting documentation for systems and models to help customers understand how they work.
• Vulnerability management—Proactively identify and patch vulnerabilities in AI systems.

3.19 AI Software Supply Chain Risk

Software supply chain risk is nothing new; however, traditional supply chain risk management addresses individual components vs. entire systems. Managing AI software supply chain risk is needed not just to reduce the attack surface or remediate vulnerabilities, but to protect an enterprise’s brand, reputation, and trust. Therefore, it is important to fully understand the AI supply chain.

Figure 3.29 illustrates the corresponding dimensions of people, processes, technologies, data, and models.

3.19.1 Emerging and Evolving Best Practices for AI Vendor Oversight

Because AI is such a rapidly evolving field, it is important to adhere to best practices for monitoring the performance of a vendor’s AI solution:

• Choose the right LLM monitoring metrics—Select metrics that reflect the capabilities and impacts of LLMs. Incorporate a mix of intrinsic and extrinsic metrics tailored to specific use cases.
• Set up effective logging, alerting, and monitoring—Combine visibility with responsive technologies to allow early detection and efficient mitigation of operational issues and security incidents. Define clear thresholds and monitoring frequencies and preestablished escalation paths for key indicators and metrics. Monitor for elements like prompt quality, output relevance, bias, sentiment, and toxicity.
• Run adversarial tests—Testing should include token manipulation, gradient descent,¹⁹⁶ and jailbreak attacks (which are adversarial prompt attacks targeting GenAI systems).
• Maintain strict data integrity and model input standards—Trustworthy AI systems provide users with a high degree of confidence in their accuracy. Regularly check data quality to detect and respond to bias, validate input, and monitor for anomalies and model drift.
• Never trust, always verify—Implement tests to ensure accuracy of outputs. Push systems to provide and cite sources to fact check.
• Perform a sanity check—Use multiple AI systems and compare results.
• Establish a gold image and then fine tune as needed—Establish a gold standard and tailor it for specific use cases. Fine tuning is typically faster (and cheaper) and can allow enterprises to leverage the existing capabilities of pretrained models.
• Balance model size with resource costs—When implementing GenAI features, consider the costs of deploying and querying models.

These best practices emphasize the importance of continuous monitoring, robust security measures, careful data management, effective interaction strategies, and practical considerations for building and deploying LLM applications.

3.19.2 Supply Chain Parties

There are a number of roles and responsibilities associated with the AI software supply chain. Figure 3.30 identifies each party, describes characteristics, and cites examples. Any mention of a product or service is purely for reference purposes and is not intended to be an endorsement or warranty of any kind.

Figure 3.31 illustrates the concepts of the AI relationship ecosystem. (Note: Parties in the example are for illustrative purposes only and are not intended to represent any enterprise or service provider.)

Visualizing these relationships can help an enterprise understand how AI solutions are built and address any concerns along the line. The depth of the parties involved in the supply chain can also help the enterprise to make an informed decision about whether or not it wants to engage with an AI provider.

3.20 Cloud Computing Risk in AI Supply Chains

When implementing AI solutions, organizations will have to choose a hosting strategy that best meets their needs and means. As with most technologies, hosting an AI solution on premises will result in greater overall control and potential for customization. However, there will be added costs related to computing resources, power, and potentially personnel. For cloud implementation, there is overall greater scalability and potential integrations. However, organizations may be at greater risk of vendor lock-in or uncontrollable cost fluctuations from providers. Figure 3.32 lists the benefits and limitations of internal vs. cloud-hosted AI solutions.

When hosting in the cloud, AI developers and those managing relationships with the applicable third parties need to consider contractual obligations and, potentially, compliance risk-related concerns. For example, if data to be leveraged for training of an AI solution is covered by privacy laws, the physical location of a cloud hosting provider may be of increased concern. Oversight of AI solution vendors is essential to ensure not only data security and privacy but also alignment with the organization’s overall AI strategy.

A foremost risk in cloud-based AI supply chains is data jurisdiction. Cloud service providers (CSPs) often operate data centers across multiple countries and regions, which may subject stored or processed data to varying legal and regulatory regimes. Organizations outsourcing AI workloads to the cloud must understand where their data physically resides and how this impacts compliance with privacy laws, data protection regulations, and contractual obligations. Failure to control or verify the data location can expose enterprises to legal liabilities, especially when handling sensitive or regulated information. It is critical that contractual agreements with CSPs explicitly address data residency requirements and compliance responsibilities to mitigate this risk.