Part F: AI Trustworthiness, Ethical, and Societal Implications

AI systems are designed so they can learn, adapt, improve, and evolve, often in unpredictable ways. As a result, it can be problematic to accurately account for AI risk. It is difficult to understand when and why AI might fail. Given that the goal of AI is to mimic human behavior in the context of numerous societal factors, risk includes much more than technical considerations (figure 1.26).

AI systems should be trustworthy at their core.⁶⁶ AI-based applications should foster, promote, and support the betterment and welfare of the user, whether an individual or an organization. Therefore, a risk-based approach must be used when employing these technologies. There are several common attributes that can be used in conjunction with one another to help organizations demonstrate trust in their AI systems (figure 1.27).

If not effectively managed, AI systems can quickly introduce harmful biases and erode confidence and trust.

Ethically designed AI solutions will help guard against unintended adverse effects, including potential human consequences (e.g., loss of privacy, threats to human life), IP concerns, and impacts on the physical environment.

The ethical use of AI is an overarching principle that focuses on developing and applying AI solutions in ways that align with the core values of society, promote social good, and minimize harm. AI solutions that do not meet these three principles may cause harm to humans or data and should not be created.

The ethical considerations of a proposed AI solution should be reviewed early in the organization’s development process. This can be done as part of a risk assessment and should be integrated into the overall change management process. The United Nations Educational, Scientific, and Cultural Organization (UNESCO) defines an ethical impact assessment (EIA) as a process that evaluates the overall design, development, and deployment of AI systems, allowing assessment of the risk before and after the system is released to the public.⁶⁷ AI solution developers should ensure that an EIA is performed in connection with any newly proposed AI solution and for any changes to AI solutions.

1.21 Responsible Use of AI Systems

Organization should focus on avoiding actions that may harm customers, third parties, and the enterprise itself. Deployment of AI can lead to gaps in responsibility, as described in figure 1.28.

RAI is a key concern for many enterprises, as the failure to use AI responsibly can result in negative consequences, including reputational damage, litigation, regulatory fines, privacy violations, and diminished employee and customer trust.⁶⁸ Employing RAI can help close some of these responsibility gaps by encouraging enterprises to actively engage in discussions around the impacts.

However, the desire to deploy RAI may not match the actual implementation of the AI solution. An enterprise’s culture and appropriate awareness and training efforts can ensure that employees understand the AI policy and acceptable use so that RAI efforts are successful. Figure 1.29 describes the elements of a successful RAI program.

Once established, it is important to monitor the maturity of a RAI program (figure 1.30).

AI solutions are subject to a variety of design and impact considerations related to ethics. When designing AI solutions, developers need to consider ethical use cases prior to project initiation. AI solutions must be designed to:

• Minimize bias and ensure fairness
• Ensure transparency and be explainable
• Promote trust and ensure safety for humans and data

1.22 Bias and Fairness

AI solutions must be free of bias to help ensure fairness and avoid unintended harmful effects on specific groups of people. AI solutions can often perpetuate or amplify biases present in the datasets they are trained on. NIST Special Publication 1270 identifies three different categories of AI bias:⁶⁹

1. Systemic—Systemic biases come from the procedures and practices of particular institutions that operate in ways that result in certain social groups being advantaged or favored and others being disadvantaged or devalued. For example, in 2018, Amazon developed an AI recruitment tool to screen top talent.⁷⁰ However, the tool exhibited significant gender bias and favored male candidates over female candidates. While this bias was partially related to the dataset itself, the algorithm amplified patterns in the data without an understanding of the ethical implications. The implementation of fairness constraints or bias audits before deployment could have helped to mitigate this issue.
2. Statistical/Computational—Statistical and computational biases stem from errors that occur when a sample is not representative of the population. For example, a 2018 study found that facial recognition systems from leading technology companies had an error rate of 0.8% for white men but an error rate as high as 34.7% for Black women.⁷¹ Improving the diversity of the dataset that the facial recognition systems were trained on could have helped mitigate this disparity.
3. Human—Human biases reflect systemic errors in human thought based on a limited number of heuristic principles and predicting values to simpler judgmental operations. For example, different users may craft prompts for GenAI solutions that lead to vastly different results and potential interpretations. This can lead to bias effects outside of the AI solutions themselves, such as anchoring bias or confirmation bias. Providing user awareness and training, as well as ensuring that the output from AI solutions is cited and explainable, are ways to avoid human bias.

Bias can lead to unfair outcomes or actions taken either by an AI itself or by users of biased AI solutions. Fairness in AI means ensuring that AI solutions promote equitable and accurate results that do not harm vulnerable or marginalized groups. Developers of AI solutions must proactively address bias to help ensure the overall trust and safety of AI systems. There are many tools and frameworks available to help developers identify and address bias during development, including:

• AI Fairness 360⁷²—An open-source toolkit to help developers of AI solutions examine, report, and mitigate discrimination and bias in ML models throughout the AI application life cycle
• What-If Tool⁷³—An open-source application provided by Google that lets developers of AI solutions test performance in hypothetical situations, analyze the importance of different data features, and visualize behavior across multiple models and subsets of input data

1.23 Transparency and Explainability

AI solutions should be transparent and easily explainable so that users can understand how decisions are made and verify their validity. If AI solutions are treated as black boxes, there is a higher risk that untrustworthy AI outputs may lead to unintended consequences. AI developers may be hesitant to provide too much information about their AI solutions, as the more information that is out there, the easier it is for hackers to find and exploit vulnerabilities. Additionally, explaining AI is generally hard, especially when trying to convey complex programs to nonexperts.

According to IBM, organizations should put clear principles for trust and transparency into practice and embed them into the entire AI life cycle.⁷⁴ Transparency can be achieved primarily through clear documentation about the AI solution. This includes sharing information about the model (e.g., name, purpose, risk level, policy, generation), the training data used, any assumed bias, fairness and explainability metrics, and contact information.

Beyond transparency, the AI solution must be explainable. The primary question that AI developers and users should be able to answer is, “How did the AI solution arrive at its result?” One way this can be achieved is by citing sources in AI outputs for users to reference.

Transparency and explainability standards and guidelines are central points for many new and evolving AI-related laws and standards. Ensuring transparency and explainability in AI leads to greater trust and improves the safety of AI-related solutions.

1.24 Trust and Safety

AI solutions must be trustworthy to perform as designed and not produce harmful outcomes. Developers must strive to create sound AI solutions that operate reliably, ethically, and securely, thereby minimizing risk to society while fostering confidence in their use. Developing AI solutions in a secure manner and considering ethical considerations will reduce the likelihood of a breach of trust and increase the safety of sensitive data.

There are numerous examples of an AI solution causing harm to people, resulting in loss of trust and abandonment of the system. Amazon’s previously mentioned recruiting AI is an example of a system that was abandoned after adverse effects on candidates caused loss of trust. While it is hard to quantify, the cost of abandoning an AI solution is likely to be high, considering sunk and ongoing costs (and in Amazon’s case, potential litigation).

While the impact of the Amazon example is relatively low, there is increasing risk related to enterprise AI solutions as their functionality continues to expand. This is especially true with respect to risk related to AI solutions connected to the physical world. Malfunctioning autonomous cars, for example, could potentially cause injury or even loss of human life. Ultimately, the organization that developed the AI solution may be liable. Risk managers should ensure AI developers manage this risk by taking precautions to ensure that risk is adequately assessed and appropriate controls are established to reduce the risk of potential harm to humans or data.

1.25 Human Rights and Societal Impact

Faults in AI solutions, such as bias and discrimination, could potentially harm humans. These faults often relate to privacy, data protection, and equality. Infringement of human rights can lead to reputational damage for AI users (e.g., companies, individuals), litigation, and related financial risk.

The EU AI Act requires a fundamental rights impact assessment (FRIA) for AI solutions classified as high risk. The goal of a FRIA is to assess the impact a high-risk AI could have on humans or society. See 3.5.2 Fundamental Rights Impact Assessment for High-risk Systems for more information.

1.25.1 Impact of AI on Society

AI creates both positive and negative impacts on society. On one hand, AI technologies have lowered costs, increased efficiencies, and enabled advancements in communication, commerce, and infrastructure that were previously not possible.⁷⁵ On the other hand, AI is disrupting industries, often resulting in jobs being eliminated or the misuse of proprietary, personal, or copyright data. Figure 1.31 describes some of these impacts on society.

Impact of AI Decisions

As AI solutions increasingly make initial decisions for consumers—from chatbots solving simple customer service questions to AI agents reviewing and processing resumes for job candidates—they affect more aspects of daily life. It is important for enterprises to understand the impact of even these basic decisions on the end user.

Impact of AI Technology Infrastructure

With increased AI usage comes the increase of data needed to train, test, and validate AI systems. This increased data use requires increased capacity to process AI queries, which has resulted in the quick growth and expansion of large data centers. Development of these centers can lead to economic opportunities for some areas, such as tax income and the creation of direct and indirect (e.g., construction) jobs.⁷⁶ However, loss of public recreation and green spaces, increased traffic from commuters, and strain on local resources have raised concerns among those living in communities surrounding these data centers.⁷⁷

Impact on the Workforce

As noted, the implementation of AI solutions has transformed the workforce, often creating a need to retrain employees or, in some cases, eliminate positions. Enterprises will need to carefully implement change management processes to help employees navigate changes in job responsibilities and roles due to AI. However, the impact of AI use may be farther-reaching than changes to job descriptions.

A recent study from Stanford University indicates that the use of AI, and GenAI in particular, has resulted in a 13% decrease in entry-level job roles in AI-exposed occupations, especially for those aged 22 to 25 years.⁷⁸ It is important to note that researchers found the biggest impact in those careers where AI automates, rather than augments, work. Some data also shows college students moving away from occupations that are greatly impacted by AI, such as computer science. While the full impact of AI usage on entry-level and recent graduates is unknown, it is important to monitor these changes and prepare for shifts in the workforce.

1.26 Environmental Impact

AI solutions are generally very resource-intensive for currently available computing components. This has led to a need for more data centers, requiring additional resources and power to keep up with the demand of AI. According to the International Energy Agency (IEA), a request made through ChatGPT consumes approximately 10 times the electricity of one use of the Google search engine.⁷⁹ Additionally, using Ireland as an example, the IEA estimated that data centers will account for nearly 35% of the country’s energy use by 2026.

Data centers are also contributing to pollution in some areas. In a study of California data center clusters, increases in diesel emissions and use of fossil fuels were found as well as impacts to public health, such as contribution to asthma-related symptoms.⁸⁰ Unsustainable consumption of global memory and storage by AI data centers are rapidly depleting supply chains and driving up costs for market stakeholders.⁸¹

Beyond the computing and energy requirements of AI solutions, there are many other environmental impacts to consider. The systems and data centers used by AI solutions require critical minerals and rare elements often extracted using unsustainable mining practices. Additionally, water consumption for the purpose of cooling has raised concerns, as AI-related infrastructure may soon consume six times more water than the country of Denmark (a population of six million people).⁸²

To address the environmental concerns of AI, more than 190 countries have adopted a series of nonbinding recommendations on the ethical use of AI.⁸³ While laws and regulations specific to the environment are still lacking, AI developers should proactively consider the effects of AI solutions on the surrounding environment and strive to make algorithms more efficient. In the long term, standardized procedures will be needed to measure the environmental impacts of AI to drive laws, regulations, and related incentives to limit AI’s environmental impact.

1.26.1 AI and Sustainability

As a result of these environmental concerns, more enterprises are looking into sustainability solutions to address environmental, societal, and governance (ESG) concerns that also affect supply chain considerations. Some organizations are looking to develop “sustainable” data centers that leverage alternative biofuels and waterless cooling systems to minimize the carbon footprint of these facilities.⁸⁴ Other companies are looking for innovative ways to scale AI processing needs through the use of new ethernet technologies to connect distributed data centers rather than building larger facilities.⁸⁵

Likewise, AI solutions have been leveraged to investigate and suggest solutions for environmental and sustainability efforts. The European Space Agency (ESA) Discovery researched how AI-equipped satellites could be used to detect methane leaks and national disasters from space, as well as create systems to detect and track certain types of plastics polluting the oceans.⁸⁶ AI has also been leveraged in agriculture to look at areas such as water management, pest control, and other aspects of crop production.⁸⁷