Part C: AI Ownership, Oversight, and Accountability

The need for clear ownership of AI models, solutions, and decisions is crucial. Following the release of ChatGPT, an expectation was that a chief AI officer (CAIO) role should be established in enterprises leveraging AI solutions. This has led to existing roles filling the gaps needed for AI oversight. Effective AI governance includes oversight mechanisms that address risk, such as bias, data drift, information security, data governance, system performance and degradation issues, privacy infringement, and misuse, while fostering innovation and building trust. An ethical, AI-centered approach to AI governance involves a diverse range of stakeholders, including AI developers, users, policymakers, and ethicists, ensuring that AI-related systems are developed, used, and continuously monitored in accordance with the values and norms of the society or jurisdiction impacted by them.

1.7 AI-related Roles and Responsibilities

Understanding AI-related roles and responsibilities is an important step for any organization developing or implementing AI solutions. Roles and their related responsibilities vary based on the size of the organization and its AI adoption strategy. For example, an organization that implements third-party AI solutions may not have development or operations-related roles in its IT department. Common categories of AI roles and their related responsibilities are shown in figure 1.18.

1.7.1 Role of the Enterprise’s Governing Body

The responsibility of the governing body to establish goals in traditional contexts extends to both financial objectives and nonfinancial concerns, including the culture, values, and ethics of the enterprise. Organizational and governance policies are usually created and applied through a combination of controls, plans of business, strategies, job descriptions, accepted practices of professional discipline, regulations, training, key performance indicators (KPIs), and a variety of executive communications. The governing body is responsible and accountable for all the activities of an organization, and this responsibility cannot be delegated. Therefore, the governing body needs to consider the implications of any new tool, technique, or technology an organization may adopt, including AI.

The members of the governing body must demonstrate to stakeholders that their policies (and related implementation plans) are in place to govern the effective delivery of the organization’s AI products and interactions via the human resources, processes, and technologies in use. In this sense, the responsibility for the introduction of AI and its consequences is not new. However, AI has the potential to allow for new organizational objectives and to meet or expand existing ones in a more effective and efficient way.

The governing body must determine if the intended use of AI is in line with its risk appetite. The risk can change quickly. New knowledge and a proactive governance system provide an organization with the means to respond to risk, such as modifying or aborting project plans, if necessary.

As they can be held legally accountable for any bad actions executed by an AI solution, members of the governing body must ensure that practices are appropriate for the specific uses to which AI is applied within the organization. This includes the review and, when necessary, the improvement of:

• Management through policies, strategies, resource allocation, ethical codes, declarations of values, purposes, or other instruments related to the use of AI within the organization
• Supervision through an evaluation of the AI solution and its value to the organization, along with its risk appetite and application guarantees; its monitoring, measurement, and sound decision-making capabilities; and other mechanisms related to the use of AI in the organization
• Evaluation of internal and external factors, such as current and future threats and opportunities, and the effectiveness and efficiency of governance-established mechanisms to explain the rationale for decisions made and options adopted
• Reporting to demonstrate to interested parties that the use of AI is being effectively governed by those responsible
• Potential oversight or final approval over change management processes related to AI use

1.7.2 Stakeholders

When integrating AI into a project, business, or organization, it is essential to consider the perspectives and needs of stakeholders, both internal and external to the enterprise.

Best practices for addressing stakeholders’ concerns include:

• Stakeholder mapping—Identify all relevant stakeholders and their specific concerns.
• Prioritize stakeholder concerns—Assess and prioritize stakeholders and their needs (e.g., risk assessment, gap analysis) to ensure concerns are addressed in a logical way that avoids the realization of risk.
• Engagement and communication—Regularly update stakeholders on progress, and address feedback.
• Ethical AI development—Implement frameworks to guide RAI development and use.
• Monitoring and evaluation—Continuously assess AI’s performance and impact on stakeholders.

By addressing these considerations, organizations can build trust, ensure success, and maximize the value of AI solutions for all involved.

Internal Stakeholders

Key internal stakeholders are described in figure 1.19.

External Stakeholders

Key external stakeholders are described in figure 1.20.

1.7.3 AI Charter

Creating a charter and establishing a steering committee for an AI initiative ensures clear governance, accountability, and strategic alignment. A charter formally defines the scope, objectives, and work of the AI project. It serves as a guiding document to align stakeholders and establish expectations.

Key components of an AI charter are shown in figure 1.21.

1.7.4 AI Steering Committee

The AI steering committee provides strategic oversight, ensures alignment with organizational goals, and resolves major issues during and after the AI project life cycle. The key responsibilities of the committee are to ensure AI initiatives maintain strategic alignment and to make major decisions related to AI within an enterprise, such as budget changes, scope adjustments, risk, and challenges.

The AI steering committee ensures that AI practices are ethical and compliant with regulations and laws. This includes reviewing policies regarding AI use, data handling, and bias mitigation.

Composition of the AI steering committee should be cross-functional and include representation from the lines of business impacted by AI projects. Typically, the committee consists of:

• Chairperson—Senior leader (e.g., chief information officer, chief technology officer, or chief data officer)
• Project sponsors—Executives funding or championing the project
• Domain experts—Representatives from key business units (e.g., operations, marketing, HR)
• Technical experts—AI/data science leaders, security experts, developers, and architects
• Compliance officers—Legal or ethical advisors to ensure AI governance

Regularly scheduled (e.g., monthly or quarterly) meetings are needed, along with ad hoc meetings for making critical decisions.

1.8 Accountability and AI

In most IT implementations, accountability is associated with the duty to explain, justify, and take responsibility for the actions and decisions made by or related to the system.⁵⁸

Additionally, several characteristics of AI make establishing accountability more challenging:⁵⁹

• Complexity and opacity—AI solutions are complex and often opaque, making it difficult to understand the underlying decision-making processes.
• Distributed responsibility—AI solutions are complex and often involve multiple systems and stakeholders. For example, system developers, data engineers, and end users may all be responsible for different outputs and components of the AI solution. This can make it difficult to assign accountability and liability for decisions.
• Regulatory gaps and challenges—The rapid evolution of AI means that advancements in technology often outpace the regulatory environment. Currently, reporting requirements for the accountability of AI decisions is limited or nonexistent. Likewise, AI solutions often operate in multiple jurisdictions, making it difficult to develop and enforce uniform and consistent accountability standards.

Structured AI frameworks can assist in this task, especially by ensuring accountability is embedded into all phases of the AI life cycle. See Chapter 2 AI Life Cycle Risk Management for more information.

1.9 RACI for AI Solutions

The use of a responsible, accountable, consulted, informed (RACI) matrix is a critical practice for clearly defining roles and responsibilities throughout the AI solution life cycle, including development, deployment, and ongoing management.

In the context of AI solutions, the RACI categories denote:

• Responsible—Individuals or teams who perform the actual work related to AI solution tasks, such as data scientists, AI developers, or risk practitioners who execute risk assessments and mitigation activities.
• Accountable—The single person ultimately answerable for the successful completion and oversight of AI-related tasks. This role often falls to senior management or designated AI governance leaders who ensure that AI risk management efforts are properly staffed, budgeted, and aligned with organizational objectives.
• Consulted—Stakeholders who provide the input, expertise, or approvals necessary for AI solution activities. These may include business process owners, compliance officers, legal advisors, or external experts who contribute to ethical, legal, and operational considerations.
• Informed—Individuals or groups who need to be kept updated on the progress, outcomes, or changes related to AI solutions but are not directly involved in the execution.

Figure 1.22 shows a sample RACI chart for an AI project. (Roles: AI Steering Committee, Chief Risk Officer, Chief Information Security Officer, Head AI Architect, Head AI Engineer, Information Security Manager, Privacy Officer.)