Transparency and Trust

In May 2017, a Michigan man named Willie Lynch was convicted of selling drugs to an undercover officer. At his sentencing hearing, the judge referenced a risk assessment score generated by a proprietary algorithm called COMPAS. The algorithm had deemed Lynch a high risk for recidivism, and the judge cited this determination as one factor in imposing a relatively harsh sentence. When Lynch's attorneys requested information about how the algorithm reached this conclusion, they were told the methodology was a protected trade secret. Neither the defendant nor the judge could examine the factors that influenced this consequential determination.

This case exemplifies what has become known as “the black box problem” in artificial intelligence. As algorithms increasingly influence or determine high-stakes decisions—from criminal sentencing to loan approvals, hiring decisions to medical diagnoses—their inner workings often remain opaque to those affected by their judgments. This opacity creates fundamental challenges for accountability, contestability, and trust. How can we evaluate whether an algorithm's reasoning is sound if we cannot understand how it reaches its conclusions? How can those subject to algorithmic judgments challenge potentially erroneous or biased decisions if they cannot see the basis for those decisions? How can society establish appropriate governance for technologies whose operations even their creators may not fully comprehend?

These questions take on particular urgency in the context of intelligence amplification. If AI systems are meant to enhance human judgment rather than replace it, humans must understand enough about how these systems work to integrate their outputs appropriately into decision processes. Without this understanding, we risk creating not genuine intelligence amplification but cognitive offloading—surrendering judgment to systems we neither understand nor can effectively oversee.

This chapter explores the challenges of transparency and trust in AI systems, examining both technical and social dimensions of the black box problem. It considers approaches to building systems people can understand and trust, from technical solutions like explainable AI to institutional practices that promote appropriate reliance. Most importantly, it examines the role of explainability in mitigating harm—how transparency can help ensure that AI amplifies human wisdom rather than merely human bias or folly.

The Black Box Problem: Understanding What We've Created

The black box problem refers to the difficulty or impossibility of understanding how AI systems transform inputs into outputs. This opacity emerges from multiple sources, varies across different types of systems, and creates distinct challenges for different stakeholders.

Technical Opacity

Technical Opacity arises from the inherent complexity of modern machine learning systems. Deep neural networks, for instance, may contain millions or billions of parameters adjusted through training processes that human observers cannot directly follow. The resulting models perform pattern recognition through mathematical operations distributed across many layers of artificial neurons, with no central decision logic that resembles human reasoning.

This architectural complexity means that even the systems' creators often cannot explain precisely why a particular input produces a specific output. They can describe the model's structure, training process, and overall performance, but cannot trace the exact reasoning path for individual decisions. This limitation differs fundamentally from traditional software, where developers can examine code line by line to understand its operation.

The language model GPT-4 exemplifies this technical opacity. Its responses emerge from statistical patterns learned across trillions of word combinations, not from explicit rules or knowledge representations. When it generates text that appears thoughtful or insightful, this results not from conscious reasoning but from complex pattern matching that mimics the statistical structure of human-written text. The apparent coherence of its outputs masks fundamental limitations in its “understanding”—a point made vividly when these systems confidently generate plausible-sounding but entirely fabricated information.

Corporate Secrecy

Corporate Secrecy compounds technical opacity when commercial interests restrict access to information about how AI systems operate. Companies frequently treat their algorithms, training data, and evaluation methods as proprietary trade secrets, limiting external scrutiny and independent evaluation.

This secrecy creates particular challenges for public oversight of systems with significant societal impacts. When algorithms influence lending decisions, healthcare resource allocation, or criminal justice outcomes, their protection as intellectual property conflicts with principles of transparency and accountability that normally govern such consequential domains.

The COMPAS recidivism prediction algorithm mentioned earlier exemplifies this tension. Despite its use in criminal sentencing—a context with strong due process requirements—its developer, Northpointe (now Equivant), refused to disclose the specific factors and weightings used in its risk calculations. This secrecy prevented defendants, attorneys, judges, and researchers from fully evaluating whether the system operated fairly and accurately.

Scale and Complexity

Scale and Complexity of modern AI deployment creates systemic opacity even when individual components might be relatively transparent. As AI systems interact with each other and with complex social institutions, their aggregate effects become increasingly difficult to predict, understand, or govern.

Social media recommendation algorithms illustrate this systemic opacity. While individual recommendation engines might operate according to comprehensible principles—promoting content that generates engagement, for instance—their collective operation within vast information ecosystems creates emergent dynamics that neither designers nor users fully comprehend. The resulting patterns of information flow, attention allocation, and belief formation exceed what any single actor can effectively model or control.

This systemic complexity means that even if we could “open the black box” of individual algorithms, we might still struggle to understand their real-world impacts when deployed at scale in dynamic social environments. Technical transparency alone doesn't guarantee systemic comprehensibility.

Cognitive Gaps

Cognitive Gaps between algorithmic and human reasoning create perhaps the most fundamental form of opacity. Even when AI systems provide explanations for their outputs, these explanations may not align with how humans conceptualize the relevant domains. The result is a form of cognitive translation problem—humans and algorithms may use the same terms but mean quite different things by them.

Medical diagnosis provides a vivid example. A doctor's understanding of “pneumonia” encompasses physiological mechanisms, patient experiences, contextual risk factors, and treatment implications. An AI system trained to identify pneumonia in chest X-rays may detect statistical patterns in pixel distributions that reliably correlate with the disease but bear no resemblance to human diagnostic reasoning. When asked to “explain” its diagnosis, the system might highlight image regions that influence its prediction without capturing the conceptual understanding that gives meaning to human diagnostic judgments.

This cognitive gap means that transparency isn't just about seeing inside the black box but about translating between fundamentally different modes of information processing. For AI explanations to be useful, they must bridge between statistical pattern recognition and the conceptual frameworks humans use to understand the world.

These forms of opacity—technical, corporate, systemic, and cognitive—create distinct challenges for different stakeholders in AI systems:

Developers need to understand how their systems function to identify and address problems like bias, brittleness, or unexpected behavior. Technical opacity limits their ability to predict how systems will behave in novel situations or to diagnose failures when they occur.

Users need to understand enough about AI capabilities and limitations to determine when and how to incorporate algorithmic outputs into their decisions. Without this understanding, they risk either over-relying on systems in contexts where they perform poorly or under-utilizing them where they could provide valuable assistance.

Subjects of algorithmic decisions need to understand the factors that influence those decisions to contest errors, address disadvantages, or simply make sense of outcomes that affect them.

Regulators and policymakers need to understand how AI systems operate to develop appropriate governance frameworks and ensure these technologies serve public interests.

These stakeholder needs highlight why the black box problem isn't merely a technical challenge but a social and political one. Transparency serves different functions for different groups, and addressing their distinct needs requires multiple approaches—from technical methods that make AI more interpretable to institutional practices that ensure appropriate oversight regardless of technical transparency.

Building Systems People Can Trust and Understand

Addressing the black box problem requires approaches that span technical design, institutional practices, and broader governance frameworks. Rather than treating transparency as a binary property that systems either have or lack, these approaches recognize different forms and degrees of comprehensibility serving different purposes across contexts.

Explainable AI (XAI)

Explainable AI (XAI) encompasses technical methods that make AI systems more interpretable without necessarily sacrificing performance. These approaches range from using inherently more transparent model architectures to developing post-hoc explanation techniques for complex black box models.

Inherently interpretable models include decision trees, rule-based systems, and certain types of linear models whose operations can be directly inspected and understood. These approaches often trade some predictive performance for clarity of operation, making them particularly appropriate for high-stakes contexts where explainability is essential for trust and accountability.

Credit scoring offers an example where interpretable models remain valuable despite the availability of more complex alternatives. Many lenders continue to use relatively transparent scoring systems that rely on clearly defined factors like payment history, credit utilization, and account age. While more complex models might marginally improve predictive accuracy, the transparency benefits of simpler approaches—allowing applicants to understand and potentially improve their scores—often outweigh small performance gains.

Post-hoc explanation methods attempt to make complex black box models more understandable without changing their underlying architecture. These techniques include:

1. Local explanations that identify which features most influenced a specific prediction
2. Global explanations that characterize a model's overall behavior across its input space
3. Counterfactual explanations that show how inputs would need to change to produce different outputs
4. Example-based explanations that illustrate model behavior through representative cases

LIME (Local Interpretable Model-Agnostic Explanations) exemplifies this approach. This technique approximates complex models locally with simpler, interpretable ones to explain individual predictions. When applied to image classification, for instance, LIME might highlight regions of an image that most strongly influenced the model's categorization, helping users understand what visual features drove the classification.

User-Centered Explanation Design

User-Centered Explanation Design shifts focus from technical transparency to effective communication with specific stakeholders. This approach recognizes that explanations must be tailored to their audiences' needs, capabilities, and contexts of use.

Several principles guide effective explanation design:

1. Relevance to the specific decision context and user needs
2. Actionability that enables appropriate responses to the explanation
3. Accessibility to users with varying levels of technical knowledge
4. Timeliness that provides explanations when they can meaningfully inform decisions

The European Union's General Data Protection Regulation (GDPR) incorporates elements of this approach in its “right to explanation” provisions. While the exact scope of this right remains contested, it establishes the principle that individuals subject to automated decisions have legitimate interests in understandable explanations tailored to their needs, not just technical disclosures meaningful only to experts.

Institutional Transparency

Institutional Transparency complements technical explainability by making organizational practices around AI development and deployment more visible and accountable. This approach recognizes that understanding AI systems requires knowledge not just of algorithms themselves but of the human decisions that shape their design, training, evaluation, and use.

Key elements of institutional transparency include:

1. Documentation of design choices, training data characteristics, performance limitations, and intended uses
2. Impact assessments that evaluate potential effects on different stakeholders before deployment
3. Independent auditing by qualified third parties to verify claims about system performance and safeguards
4. Incident reporting that discloses significant failures, unintended consequences, or harmful outcomes

The algorithmic impact assessments required by Canada's Directive on Automated Decision-Making exemplify this approach. Government agencies must evaluate the potential impacts of automated decision systems before deployment, with increasing transparency and oversight requirements for systems with higher potential impact on rights, health, economic interests, or other significant concerns.

Trust-Promoting Interaction Design

Trust-Promoting Interaction Design focuses on how AI systems communicate with users about their capabilities, limitations, and confidence levels. This approach recognizes that trust isn't simply about technical transparency but about appropriate reliance based on accurate understanding of system behavior.

Well-designed interactions should:

1. Clearly communicate what the system can and cannot do
2. Indicate confidence levels for different outputs
3. Highlight potential error modes and their consequences
4. Provide mechanisms for questioning, correcting, or overriding system outputs

Weather forecasting apps exemplify this approach when they present precipitation predictions with explicit probability estimates rather than binary claims. This presentation helps users calibrate appropriate trust—high confidence for imminent predictions in stable conditions, lower confidence for distant forecasts or volatile weather patterns.

By contrast, many consumer AI systems encourage overconfidence through interfaces that present outputs with uniform certainty regardless of underlying confidence. Chatbots typically present generated information without indicating confidence levels, potentially leading users to trust speculative or hallucinated content as much as well-established facts.

Multi-Stakeholder Governance

Multi-Stakeholder Governance approaches recognize that no single form of transparency serves all legitimate interests in AI comprehensibility. Instead, these approaches establish governance frameworks that balance multiple considerations—including proprietary interests, privacy protections, and security concerns—while ensuring appropriate oversight for consequential systems.

These frameworks might include:

1. Tiered disclosure requirements based on application risk levels
2. Confidential access for qualified reviewers while protecting legitimate proprietary interests
3. Aggregate reporting that provides societal oversight without compromising individual privacy
4. Participatory governance that includes affected communities in oversight processes

FDA regulation of medical algorithms exemplifies this approach. High-risk medical AI systems undergo rigorous pre-market review that balances the need for thorough evaluation against legitimate protection of intellectual property. The review process includes detailed examination of validation methods and performance data without necessarily requiring full disclosure of proprietary algorithms to the public.

Together, these approaches—technical explainability, user-centered explanation design, institutional transparency, trust-promoting interaction, and multi-stakeholder governance—provide a more comprehensive framework for addressing the black box problem than purely technical solutions alone.

The Role of Explainability in Mitigating Harm

Beyond its technical and institutional dimensions, transparency serves a crucial ethical function: it helps prevent, identify, and address harms that might otherwise remain invisible or unaddressed. This harm mitigation function operates through several distinct mechanisms, each addressing different risks associated with black box decision systems.

Enabling Meaningful Contestation

Enabling Meaningful Contestation represents perhaps the most fundamental way transparency mitigates harm. When individuals understand the basis for decisions that affect them, they can identify errors, challenge flawed assumptions, provide relevant additional information, or appeal to considerations the system might have overlooked.

The case of Robert Julian-Borchak Williams illustrates this dynamic. In January 2020, Williams was arrested in Detroit based on a facial recognition system's incorrect match to surveillance footage of a shoplifting suspect. Only when shown the surveillance image during interrogation could Williams demonstrate the obvious mismatch, pointing out, “This is not me.” Had the system's role remained hidden, Williams might have had greater difficulty contesting his wrongful arrest.

Detecting and Addressing Bias

Detecting and Addressing Bias becomes possible when we can examine how systems operate across different populations and contexts. Transparency enables the identification of disparate impacts that might otherwise remain invisible, particularly when these impacts affect marginalized groups.

The Gender Shades project, led by Joy Buolamwini and Timnit Gebru, exemplifies this function. By testing commercial facial analysis systems on a demographically diverse dataset, the researchers demonstrated that these systems performed significantly worse for darker-skinned women than for lighter-skinned men—disparities that weren't apparent from aggregate performance metrics. This transparent evaluation spurred companies to address these biases in subsequent versions.

Preventing Automation of Harmful Practices

Preventing Automation of Harmful Practices by exposing them to public scrutiny and ethical evaluation. When decision processes remain hidden within proprietary algorithms, practices that would generate public outcry if explicitly acknowledged can continue under the guise of neutral, objective computation.

HireVue's now-discontinued practice of analyzing candidates' facial expressions during video interviews exemplifies this dynamic. The company claimed its algorithms could assess candidates' employability by analyzing subtle facial movements during recorded interviews. Only when this practice faced public scrutiny did its questionable scientific basis and potential discriminatory impact become widely discussed, eventually leading to its abandonment.

Enabling Proper Attribution of Responsibility

Enabling Proper Attribution of Responsibility by clarifying the relationship between human and algorithmic decision-making. When algorithmic systems operate as black boxes, responsibility for harmful outcomes can become diffused or displaced, with humans blaming algorithms and algorithm developers blaming human misuse.

The case of Dutch childcare benefits scandal illustrates this danger. Between 2013 and 2019, a partially automated fraud detection system falsely flagged thousands of families—disproportionately those with immigrant backgrounds—as having committed fraud against the childcare benefits system. These false accusations led to severe financial hardship, home repossessions, relationship breakdowns, and even suicides among affected families. The system's opacity contributed significantly to this harm, as officials couldn't effectively evaluate its accuracy and affected families couldn't understand why they'd been flagged.

Preserving Human Agency and Wisdom

Preserving Human Agency and Wisdom by preventing excessive deference to algorithmic recommendations. When systems operate as inscrutable black boxes, humans often exhibit automation bias—the tendency to give automated systems greater authority than warranted, particularly in areas where they lack confidence in their own judgment.

Medical diagnostic systems demonstrate both the promise and peril of this dynamic. Studies show that AI systems can identify certain conditions from medical images with accuracy comparable to expert radiologists. However, these systems typically analyze images in isolation, without the patient history, physical examination findings, and clinical context that human physicians integrate into their assessments.

When these systems operate transparently—clearly communicating what they're evaluating, what patterns they're detecting, and what limitations they face—physicians can appropriately integrate their recommendations with broader clinical judgment. Transparency thus serves not just technical accountability but the deeper goal of genuine intelligence amplification.

Enabling Democratic Governance

Enabling Democratic Governance of increasingly powerful technologies that shape social outcomes. In democratic societies, citizens have legitimate interests in understanding and influencing how consequential technologies operate. When these technologies remain opaque, meaningful democratic oversight becomes impossible, effectively transferring power from democratic institutions to technical systems and their creators.

The governance of social media recommendation algorithms exemplifies this challenge. These systems significantly influence information exposure, belief formation, and civic discourse, yet they operate largely without transparent explanation or democratic accountability. Their optimization for engagement rather than civic health or democratic values has raised significant concerns about effects on political polarization, misinformation spread, and democratic deliberation.

These multiple functions of transparency in harm mitigation highlight why the black box problem isn't merely a technical challenge but a profound ethical and political one. This perspective suggests that we should approach transparency not as a technical feature to be maximized uniformly across applications but as a contextual requirement whose importance varies with:

1. The stakes and consequences of the decisions involved
2. The potential for harm to vulnerable populations
3. The importance of contextual judgment and ethical considerations
4. The centrality of the application to democratic governance and public values

As we design, deploy, and govern increasingly powerful AI systems, ensuring appropriate transparency represents one of our most important safeguards against unintended harm. By enabling meaningful contestation, bias detection, proper responsibility attribution, calibrated trust, and democratic oversight, transparency helps ensure that AI amplifies human wisdom rather than merely human bias or folly.

The path forward requires both technical innovation in explainable AI and institutional commitment to transparent governance. It demands recognition that transparency isn't just a technical feature but a social relationship—a commitment to making powerful technologies understandable to those whose lives they affect. Most fundamentally, it requires acknowledging that technologies that cannot be meaningfully understood by those who create, use, and are subject to them should not be deployed in contexts where significant harm might result from that lack of understanding.

By keeping humans “in the loop” not just as nominal decision-makers but as informed, empowered participants who genuinely understand the systems they oversee, we can work toward AI that truly enhances human capability rather than merely displacing human judgment. This vision of intelligence amplification—human and machine capabilities complementing rather than replacing each other—offers our best hope for harnessing AI's potential while mitigating its risks.