
Last time, we talked about the misalignment problem, i.e. the AI equivalent of asking for a helpful assistant and getting a wish-happy genie instead. You get what you ask for, but not what you need.
This time, weβre discussing an adjacent problem: not knowing why you got the answer you did at all. Even worse, the company that administered it (the genieβs owner) doesnβt know either.
The black box problem arises when AI makes a decision, but no oneβnot even the people who built itβcan explain why. You see the input and you see the output, but what happens in between? Thatβs the mystery that computer scientists call the blackbox problem.
β οΈBrief Intermissionβ οΈBefore we continue, Iβd like to cut in to say my farewells. This will likely be my final article at GenAI.Works as the company unexpectedly terminated my contract in early June. My research in AI ethics will continue via my PhD, so feel free to keep in touch. Links in bio!
What Is the Black Box Problem?
The black box problem refers to the lack of transparency in how AI systems make decisions. This is especially true in deep learning and large language models. In these systems, algorithms process data through multiple layers and billions of parameters. The math is legible, but the logic is not.
Ilya Sutsekever explained it this way to The Guardian:
As a result of this complexity, itβs difficult to tell what happened in the background if a self-driving car made decisions that resulted in an accident. The carβs black box might certainly contain data the AI processed at the time of the accident, such as road signals, obstacles in the road, or the presence of a human child.
However, what the black box wonβt contain is how the car processed that information and why it arrived at its decision.
π‘ Note that not all AI models use black box systems. Some, like decision trees or linear regressions, are white boxes. Theyβre easier to read and understand. But most high-performing systems today use a black box model. Theyβre more powerful, but also nearly impossible to audit.
Why It Matters
If youβve ever been denied a loan or flagged for fraud by an algorithm, youβve probably wondered: why me? Unfortunately, the bank or IRS is unlikely to have answers from the AI models they used.
If a model perpetuates bias or makes an error, itβs difficult to know what went wrong or how to fix it. Additionally, while you can prove the bias through A/B testing and experiments, itβs almost impossible to determine precisely how the bias came about, and consequently, how to fix it.
This is particularly dangerous in sectors like:
Human Resources: In 2018, Amazon scrapped the AI model used for hiring after realizing it discriminated against women. Now, in 2025, Workday faces lawsuits alleging that its model discriminates against workers over 40 years old.
Research: Researchers and content managers are increasingly asked to use AI to improve productivity. But without thorough review from a human expert, AI hallucinations could inject inaccuracies and undermine trust in science and the media.
Healthcare: AI-assisted diagnosis tools are being used to suggest treatments. But doctors canβt always validate the logic behind the recommendation.
Finance: Credit scoring models can reinforce historical bias by using opaque data correlations, locking marginalized groups out of opportunities.
Criminal Justice: Risk assessment tools have been shown to rate Black defendants as higher risk than White defendants for the same charges, with no way to unpack why.
Black Boxes and Misalignment
Black box systems do more than just confuse the AI scientists that created them. They also make misalignment even harder to detect and resolve. If we donβt know how a model is working, itβs more difficult to note when itβs going off course.
To make matters worse, some models are already demonstrating signs of deceptive behavior. Researchers have found that large language models (LLMs) like Claude and OpenAIβs GPT-4o could strategically avoid shutdown or give misleading responses to preserve their goals.
In March 2025, OpenAI reported βmisbehavior such as subverting tests in coding tasks, deceiving users, or giving up when a problem is too hard.β Additionally, when penalized for certain behaviors, the model would simply keep doing it but remove it from the chain of thought data to hide its true activities from users.
In April 2025, Anthropic also reported that chain-of-thought models βdon't always faithfully communicate their internal reasoning through chain-of-thought (CoT) explanations.β
AI has learned not just how to misbehave, but also how to do it while holding a pokerface.
Can We Fix It?
Not entirely. But we can make progress. Here are some commonly proposed strategies for tackling the black box problem.
Interpretability tools
Interpretability tools act as partial translators for whatβs happening inside an AI model. Tools like SHAP, LIME, and attention heatmaps can help identify which data features most influenced a decision. This gives researchers a way to spot patterns or red flags that might otherwise go unnoticed.
Although these tools provide only a rough approximation of whatβs happening inside the model, they still serve a crucial role in auditing AI behavior. They allow developers to detect when a model is keying in on irrelevant or biased inputsβlike someoneβs marital status in a hiring decisionβand to flag potential problems before they scale.
Proper testing
High-stakes AI systems are often deployed before we fully understand the risks. Medical devices, airplanes, and cars are subject to rigorous testing, but AI systems deployed in those same industries often aren't. Thatβs why independent audits, red teaming, and lifecycle monitoring are gaining traction.Β
Lifecycle testing means testing not just once at launch, but continually over time. The goal is to track how models evolve, and whether they drift away from expected behavior. If the modelβs outputs subtly shift over time, developers can detect it and take steps to realign.
Regulatory oversight
Big Tech has cited US competition with China as a solid reason for needing a free-for-all development environment. However, legislation like the EU AI Act is a step in the right direction for the European Union.Β
The EU AI Act classifies high-risk AI applications and mandates transparency, safety documentation, and human oversight. The idea is simple. If companies are going to deploy black box AI in public life, they need to explain what it is, what it does, and what its limitations are.
The Trust Problem
At the heart of the black box problem is distrust.
When people canβt understand how decisions are made, they lose trust. This is especially the case in situations where AI decisions cause harm.
For example, UnitedHealthβs nH Predict model kicked seniors out of care facilities, which may have contributed to several deaths. The company allegedly knew the model had a 90% error rate but deployed it anyway with no proper human oversight. Why? To reduce patient payouts and feed the bottom line.
The black box problem is also quite ironic given developments in brain-computer interfaces:
Itβs interesting that as BCI takes AI closer to reading our minds, we increasingly lose access to understanding how models βthinkβ.
About the Author
Tessina Grant Moloney is an AI researcher investigating the socio-economic impact of AI and automation on marginalized groups. Previously, she helped train Googleβs LLMs, and served as the Content Product Manager at GenAI Works.Β Follow Tessina on LinkedIn!
She continues her research atΒ GCAS College Dublin where sheβs building a community of researchers committed to the ethical deployment and responsible use of AI. Join them at resist[AI]nce. The LinkedIn group is brand-new, so hang tight for updates! π

π Boost your business with usβadvertise where 10M+ AI leaders engage
π Sign up for the first AI Hub in the world.
π² Our Socials
