Column
"Are you an AI?" Analyzing Client Suspicion of AI Use in Crisis Counseling
In a crisis counseling service with no AI involvement whatsoever, client suspicion of AI use rose from 0.8% in June 2024 to 2.6% by March 2025. Suspicion concentrated in the early conversation phase, and even when counselors denied AI involvement, 17.6% of cases ended with suspicion unresolved or conversation terminated. AI distrust can erode therapeutic relationships long before any AI is deployed.
Hello. I’m Inoshita from the Affectosphere Group.
“Are you really human?”
When someone in crisis asks this of their counselor, it is not a simple verification request. It is a challenge to the very foundation of trust.
A study published on arXiv (Shah et al., arXiv:2606.18261) tackled a striking paradox: a WhatsApp-based crisis counseling service in India uses no AI whatsoever, yet client suspicion of AI involvement kept rising. The research analyzed 75,777 conversations collected between June 2024 and March 2025.
AI distrust emerges independently of whether AI is actually deployed. This fact raises a challenge that everyone designing and implementing affective AI must face directly.
Three Key Points
- AI suspicion occurs and increases even when no AI is involved.
- Suspicion concentrates in the first half of conversations (68.3%), and even when counselors deny AI involvement, 17.6% of cases remain unresolved or end prematurely.
- 21.5% of clients expressing suspicion explicitly state they want to speak with a human, fundamentally undermining the therapeutic relationship.
The Numbers Behind “Growing AI Distrust”
The proportion of conversations where AI suspicion was identified rose from 0.8% in June 2024 to 2.6% by March 2025 — more than tripling in roughly nine months.
What drives this? The study does not offer definitive answers, but the broader societal shift seems relevant. As AI awareness has grown and media coverage of AI-driven text communications has expanded, people naturally become more alert to the question of whether any given text was written by a machine.
The nature of crisis counseling amplifies this dynamic considerably. When someone is in acute distress, they intensely need to feel that their pain is genuinely understood by a real person. The suspicion that their listener might be artificial carries consequences far more severe than in ordinary customer service interactions.
Suspicion Hits at the Start
A particularly consequential finding concerns when suspicion emerges during a conversation.
68.3% of cases show suspicion arising in the first half of the interaction.
Clients are evaluating whether to trust their conversational partner from the very first exchange — often before they have shared anything deeply personal. Many continue talking while already harboring doubt.
In counseling practice, building rapport in the first few minutes is considered critical. Suspicion concentrating precisely in that opening phase means counselors face a structural disadvantage from the moment a conversation begins.
Response speed, phrasing patterns, and the texture of empathetic expressions may all contribute to triggering the perception of “AI-ness.” This suggests that how a counselor communicates — not just what they say — plays a role in whether suspicion takes hold.
”I Want to Talk to a Human”
Among clients who expressed suspicion, 21.5% explicitly stated they wanted to speak with a human. Explicitly — meaning they put it into words.
Someone in crisis, reaching out for help, simultaneously asking whether their helper is real and requesting a human alternative. This dynamic introduces friction at the very moment connection is most needed.
The other critical figure: even when counselors denied AI involvement, 17.6% of cases resulted in continued suspicion or conversation termination.
Denial does not reliably resolve the distrust. In some cases, the conversation ends before a denial can even be heard.
This points to something important: a simple informational response (“I’m a human, not an AI”) cannot overcome distrust that is rooted in the felt quality of the interaction itself. What is being interrogated is not a factual claim but the gestalt of the exchange — something that must be conveyed through the entire arc of communication, not a single reassurance.
A Problem That Predates Deployment
This research illuminates a domain that tends to be overlooked in affective AI design discussions.
Most AI deployment conversations center on how to design systems, how to evaluate their performance, and how to build trust in AI outputs. What this study reveals is that users develop AI-related distrust based on their own perception — regardless of whether AI is actually present.
This creates a meaningful asymmetry.
No matter how human-like an AI is designed to be, once a user perceives it as artificial, trust collapses. Conversely, human counselors who communicate in ways perceived as “AI-like” generate suspicion even with no AI involved.
For those designing and deploying affective AI systems, two questions follow directly.
The first is an interaction design question. What response patterns generate the perception of “AI-ness”? Response speed, syntactic patterns, the repetitiveness of empathetic formulas — understanding which elements trigger suspicion matters for designing both AI agents and human operator interfaces.
The second is a disclosure design question. When asked “Are you an AI?”, what is the right response? Simply answering yes or no is insufficient, especially under conditions of high emotional load. Context-sensitive disclosure strategies — ones that acknowledge the question seriously rather than deflecting — deserve careful design attention.
Beyond the Paradox of AI Distrust
AI suspicion without any AI present. This paradox signals that “AI distrust” has become a feature of social cognition, not merely a response to specific technology.
Since the widespread adoption of large language models, a “could this be AI?” lens has been built into how people read text communications. Readers unconsciously assess the “AI-ness” of what they encounter.
This shift appears irreversible.
If it reaches even the most intimately human settings — crisis support — then those of us working with affective AI must design with that baseline assumption in place. Not “how do we build trust” but “how do we design interactions that can resolve suspicion when it arises.”
That is the research and practice direction this moment demands.
Reference
- Shah, S., Swaminathan, A., Simhadri, M., Lopez, I., Phadke, S., Verma, D., John, A., Zhao, L., Cai, F., Zhang, S., Ye, G., Pham, I., Wang, W., Garcia, S., Wornow, S., Wang, A., & Shah, N. H. (2026). “Are you an AI?” Analyzing Client Suspicion of AI Use in Crisis Counseling. arXiv preprint arXiv:2606.18261.
* This article was written in part with AI assistance and may contain inaccuracies.