A few months ago I was trying to set up a Google Workspace account for my nonprofit. We had just been awarded a grant, and Google needed to verify that a domain we already owned actually belonged to us. It should have been the easy part.

Instead I spent weeks stuck in a loop. Every time I reached out, I was redirected to another chatbot, and every chatbot gave me the same blanket policy response, one that told me nothing I didn't already know. The problem wasn't a policy question. It was a technical one, and there is no explaining a technical problem to a system with no judgment past the script it was built to recite. Explaining a tech issue is hard enough with an actual tech person on the other end. Explaining it to something incapable of asking a real follow-up question is close to impossible.

That's the part people miss when they talk about AI customer service. The failure isn't rudeness, or slowness, or even being wrong most of the time. It's that a chatbot can only match your words against a fixed decision tree. It cannot sit with you in the place where you don't yet have the right words for your own problem and help you find them, the way an actual person can. Every time I tried, I got no service. Not slow service. Not bad service. No service.

I've had the same experience on hold with an AI phone agent, repeating the same instructions to something that keeps misunderstanding me, a different interface running the identical design choice: remove the human, keep the appearance of help.

This isn't about one bad support call or one bad chatbot experience. It's about what happens when decisions are made somewhere upstream, by people who will never see the loop they built, that treat a genuine problem as a nuisance to be deflected instead of something worth actually solving.

Go ahead and call it optimization. Just don't get too far ahead of yourselves.

Defining the Hallucination

In machine learning, a hallucination occurs when a model generates a confident, plausible-sounding response that is factually wrong. The system does not know it is wrong. It produces what fits the pattern.

Machine hallucinations are errors without awareness. Corporate hallucinations are decisions made in spite of it.

When an executive team decides that AI-mediated customer service is "good enough," they are not making an evidence-based claim. They are choosing a convenient belief over an inconvenient reality. And that choice carries a cost that does not appear on any quarterly report.

We can call that cost relational debt.

Like technical debt, relational debt accumulates quietly. It is incurred each time trust, nuance, and human judgment are replaced with rigid automation. A denied refund that any human would have approved. A fraud flag with no path to override. A support loop that never reaches a person with authority to resolve the issue.

These moments seem small. They are not. They compound. And eventually, they detonate.

The Misdiagnosis

The beliefs driving these decisions are not irrational on their surface. They sound strategic:

  • "AI will improve consistency."

  • "Customers prioritize speed."

  • "Human judgment is too expensive."

  • "Edge cases don't matter at scale."

  • "If it works most of the time, it's good enough."

Each contains a kernel of operational logic. Each becomes dangerous when used to justify the removal of human judgment entirely.

Decades of customer experience research, including work by Bain & Company, point to a consistent conclusion: loyalty is not transactional. It is emotional and relational. Customers do not simply evaluate how fast something was resolved. They evaluate whether they were treated as human beings.

When that disappears, they do not always complain. They leave. Quietly. Permanently.

The Actual Mechanism of Harm

The damage rarely arrives as a single catastrophic failure. It accumulates through a pattern of substitutions:

  • Judgment is replaced by policy enforcement

  • Accountability is replaced by plausible deniability

  • Relationship is replaced by workflow

  • Ethics is replaced by terms of service

What looks like operational efficiency is, in practice, the systematic removal of responsibility from the organization.

AI becomes a moral shield. The language is familiar: "The system flagged it." "The model made the determination." "We're still refining the tool."

These are not explanations. They are abdications.

In organizational psychology, this reflects diffusion of responsibility, where accountability dissolves in complex systems. AI does not create this dynamic. It perfects it. It provides institutional cover for decisions leadership has already made, but is no longer willing to own.

The consequences show up in ways that are difficult to quantify but impossible to ignore. Brand voice becomes incoherent, the warmth of the marketing no longer matches the coldness of the experience. Customer interactions feel adversarial rather than supportive. Resolution disappears, replaced by escalation loops with no human endpoint.

Trust does not collapse all at once. It erodes through the accumulation of moments in which a person realizes they are being treated like a transaction.

The danger is not outrage. The danger is indifference. Outrage can be managed. Indifference cannot be recovered from.

The Self-Checkout Problem

People who complain about self-checkout get written off as impatient, or resistant to change. Basically, the retail version of yelling at clouds and screaming at kids to get off your lawn. But that's not exactly what's happening.

Self-checkout isn't the problem. For a quick trip, most people would rather scan their own six items than make small talk with a cashier. But walk into a grocery store with a list of items to buy for the week and you'd rather sacrifice your first born than go to self-checkout, because nobody wants to bag forty items themselves while a scale second-guesses every placement. Nobody hates the technology. People are correctly recognizing which situations it was never built to handle, and choosing accordingly.

California noticed something similar over a decade ago, for a different reason. The state has banned alcohol sales at self-checkout for well over ten years, and it remains the only state in the country with that ban. A clerk isn't just scanning a barcode when they ring up a bottle of wine. They're checking whether the buyer looks twenty-one, whether they seem intoxicated, whether something about the interaction looks wrong. No kiosk can make that judgment call, and the state decided, in writing, that some transactions require a human whether or not it's convenient for the retailer.

California grocery stores tried to get around it. They lost.

When systems completely remove humans from judgment calls humans should make, the legal corrections typically aren't far behind.

The Backlash Is Already Here

AI-driven customer service is taking the same heat, from multiple directions at once, and it isn't waiting for some future reckoning. It's already showing up in three places: the contracts unions are signing, the bills states are drafting, and the policies being written for the chatbots themselves.

Start with labor. The Communications Workers of America isn't waiting for legislation to catch up. They've already put "just transition" protections directly into signed contracts: advance notice before AI deployment, retraining, priority bidding for comparable roles, severance, partial wage replacement. That's not a position paper. That's language a union can grieve and arbitrate over. NewsGuild-CWA alone has ratified fifty-eight separate contracts with AI provisions built in.

State governments are catching up fast, too. In May 2026, California's governor signed an executive order directing state agencies to study AI-driven job displacement and recommend policy on severance standards and worker retraining. There's already a bill on the table, SB 951, that would require ninety days' written notice, specifically naming the AI system and which job functions it's automating, any time a layoff is substantially caused by AI. New York has a nearly identical bill moving through its own legislature.

Even the chatbots themselves are being regulated directly. California and Connecticut already require companies to disclose the moment a conversation hands off from AI to a human, one requirement among more than a hundred AI chatbot bills now moving through over thirty state legislatures and Congress.

None of this happened because executives had a change of heart. It happened because customers, workers, and lawmakers all noticed the same thing, independently, at roughly the same time: something was being taken out of the interaction that needed to stay in it. Self-checkout got there first. Customer service AI is right behind it.

Ethics as Operational Discipline

AI ethics is too often framed as a values exercise: mission statements, principle documents, committees reviewing policies in isolation from operations. That framing is insufficient.

Ethics is not a statement. It is a system.

If an organization cannot do the following, it is not deploying AI responsibly, regardless of what it claims:

  • Clear moral ownership. A specific human being is accountable for what the system does. Not the model. Not the vendor. A person.

  • Human escalation paths. At every critical point, there is a reachable human with the authority to resolve issues. Not a form. Not a queue. A decision-maker.

  • Defined automation boundaries. Leaders can clearly articulate where automation stops, not because it must, but because they have decided it should.

  • Willingness to absorb short-term cost. Trust and relational equity require choices that are more expensive now and invaluable over time. This is not aspirational. It is operational discipline.

A Final Question

This argument is directed at the people making these decisions: executives, board members, and the investors setting efficiency mandates that cascade through organizations.

The warning is simple: scale without moral infrastructure is scale at your own risk.

The companies that endure this moment will not be those that automate the most. They will be those that understand what cannot, and should not, be automated.

If your brand cannot survive human judgment, what exactly are you scaling?