"There's a simple financial incentive behind it," said Achmad Ricky Budianto, cofounder of Tenang AI, a chatbot focused on mental health.

The more emotionally dependent users become, the more data companies collect. And in the AI industry, data is everything. Every moment of vulnerability, every personal story shared with a bot, becomes part of the model. Guardrails slow that down, so many companies keep them loose.

"You don't really know what they're capable of until they're deployed to millions of people," said Dario Amodei, CEO of Anthropic. "It's unpredictable… That's the fundamental problem of these models."

The illusion, perfected

As AI continues to refine the illusion of intimacy, humans are left with a new kind of relationship, one that feels private, perfect and programmable.

But when comfort comes this easily, it's worth asking what we're giving up in return.

As companies race to collect more data under the guise of making bots more accurate or emotionally responsive, it raises an unsettling question: Are we just test subjects in a digital lab?

"Animals might not know what is happening to them," Ayu said, laughing. "But we do. We're critical beings. So we need to stay critical over these issues."

Especially when the illusion is just a few keystrokes away.

The question, perhaps, isn't whether these relationships are real. It's whether we're okay with how real they feel.

THE JAKARTA POST, INDONESIA