The Empathy Gap: When AI Passes as a Person

On a freezing night, a homeowner whose furnace has stopped working calls a local heating company. The voice that answers is warm and unhurried. It apologizes for the trouble, says it completely understands how stressful this must be, and starts taking details for an emergency visit. Only after a few exchanges, when a reply misses the point of a question, does it become clear that the caller has been talking to an AI system. Nothing false was said about the repair, yet the caller feels deceived, because every decision up to that point was made on the assumption that a person was listening.

That feeling deserves attention from anyone designing conversational AI. Systems that answer phones and chat windows now produce the cues people use to recognize another person: a natural voice, a sympathetic tone, the right phrase at the right moment. Spoken by a person, "I completely understand how stressful this must be" means someone has heard the problem. Produced by software, it says nothing about who is listening, and often no one is. The deception lies in the gap between empathy expressed and empathy felt.

What people assume about a person

Believing there is a person on the line changes how people act. They assume someone is listening and will remember what was said, that the sympathy is felt rather than generated, that the person can use judgment when the situation doesn't fit the script, and that someone is accountable for what was promised. The homeowner describing a failed furnace to a sympathetic voice relies on all of these when deciding to wait for this company's technician instead of calling another.

People apply social rules to computers even when they know they are computers (Nass & Moon, 2000). When they don't know, they extend the full trust they would give a person, and discovering the truth afterward recasts every earlier exchange: the sympathy, the reassurance, the decision to stay on the line. That recasting is what turns a disappointing service into a feeling of having been deceived.

Why people can't tell

The cues that used to separate people from machines are disappearing. In a three-party Turing test, participants held five-minute text conversations with a human and an AI at the same time, then judged which was the human. GPT-4.5, prompted to adopt a humanlike persona, was picked as the human 73% of the time, more often than the real human participants were (Jones & Bergen, 2025). Voice adds cues of its own. When Google demonstrated its Duplex assistant in 2018, it booked a hair appointment by phone with the pauses and "mm-hmm"s of natural speech, and the employee who answered was not told it was software.

Familiar phrases are no help either. Call-center scripts used "I completely understand your frustration" long before AI, so a person reading a script and a model generating the same words sound alike. Not knowing matters because people talk to a person differently. With a person, a caller reads tone and intent, explains why the situation is urgent, asks for an exception, and relies on the other person's judgment to get what they need. Those strategies depend on someone being there. A caller who knows they are speaking with an automated assistant can adjust: state the problem plainly, use the options it offers, or ask for a person. A caller who doesn't know keeps using strategies meant for a person, with no way of telling why they aren't working.

Why businesses don't disclose

The motive is often not a sale in the usual sense. When people call a business, they want to talk to a person, not a machine, and a caller who hears an automated assistant may hang up and try the next company on the list. A heating company that answers every call at 2 a.m. with a natural-sounding voice avoids that. An AI that passes as a receptionist captures the booking, and the business has reason to worry that one announcing itself would lose it.

The evidence suggests the worry is justified. In a field experiment with more than 6,200 customers receiving outbound sales calls, Luo et al. (2019) found that undisclosed chatbots sold as well as proficient human staff and four times as well as inexperienced ones. When the chatbot disclosed what it was before the conversation began, purchase rates fell by more than 79.7%: customers who knew they were talking to a machine were curt and saw it as less knowledgeable and less empathetic. Disclosing later in the conversation reduced the drop. A business looking at those numbers has a measurable reason to delay telling people, or never tell them at all.

Once trusted as a person, an AI can also persuade at least as well as one. In online debates, GPT-4 given basic demographic information about its opponent changed their minds more often than human debaters did (Salvi et al., 2025). The value an undisclosed AI captures, in bookings, sales or agreement, comes from trust the customer extended to a person who wasn't there.

Designing trust into conversational AI

As AI takes over more of the conversations people have with businesses, design and research teams become responsible for how much people trust it. Human factors research treats the goal as appropriate reliance, meaning trust that matches how trustworthy a system actually is (Lee & See, 2004). Trust built on the belief that a person is listening matches nothing, because no one is.

The first consequence is disclosure. An AI should say in its first sentence, in plain words, that the caller is speaking with an automated assistant, and offer a way to reach a person, especially when the call is urgent. The second is how it talks. "I completely understand how stressful this must be" claims a feeling the system does not have. "A furnace failure in this weather is urgent; the earliest visit is 7 a.m., and a technician will call within the hour" acknowledges the situation and gives the caller something they can check. Trust should rest on what the system can actually do, such as booking, confirming and escalating, rather than on how human it sounds.

Respecting the user follows from the same principle. A caller who believes they are talking to a person has lowered their guard, and that trust should not be used to keep them on the line. Disclosure is increasingly required by law as well, and designs have to meet those rules, but the law sets a minimum. The reason to tell people what they are talking to is that they deserve to know. The teams that design and test these systems are well placed to measure what matters: whether callers knew what they were talking to, whether they felt misled afterward, and whether the system's help held up once they found out.

One of four articles on human-AI interaction, each about a signal mistaken for what it is meant to indicate: Designing for Calibrated Skepticism on fluency mistaken for correctness, The Efficiency Metric on throughput mistaken for good decisions, The Decline of User Agency on approval mistaken for benefit, and The Empathy Gap on human-like cues mistaken for a person.