Instinct's founder says AI agent can hallucinate what looks like someone else's private data

If your personal AI assistant sends you what looks like intimate details about someone else's life, chances are it made up the whole thing.

At least, that's what Instinct's creator, Noah Shinn, said happened when a user of the viral AI tool said an agent admitted to leaking someone else's conversation into their chat.

The poster, @prit4k, uploaded screenshots of their interaction with Instinct on Tuesday, which show that the assistant seemingly gave them someone else's middle name and discussed a photo they never sent of a document mentioning money.

Then, Instinct apologized and straight up told @prit4k that the photo "got crossed into" their conversation, calling it a "serious glitch."

"Was this a data leak or a hallucination?" @prit4k wrote in alarm on X.

Instinct just admitted it leaked someone else's data to me It described a financial document with details that weren’t mine, then claimed someone else’s photo had crossed into my chat.But I never sent a photo@noahrshinn, was this a data leak or a hallucination? pic.twitter.com/nev6t7w85k

Shinn responded that this instance was just a hallucination. The agent had made up a proper noun and "further amplified" the mistake with its reasoning, he said.

"This was not a data breach, no user data was shared, and no user isolation boundary was violated," Shinn wrote in a post on Wednesday evening.

The 23-year-old did not say how he knew that @prit4k's case was a hallucination, but touted Instinct's privacy measures.

"We've done a lot of work to make sure that users' data are truly partitioned, from isolated sandboxes to short-lived local credentials to identity-signed tool execution," he wrote.

Add BI in Google so our reporting is easier to find when you’re searching for what matters.

Shinn did not immediately respond to a request for comment sent outside regular business hours by Business Insider.