Three limits of today’s AI: confidence, decisions and language

What the three studies tested

Three new preprints examine different AI limitations. One looks at whether people accept false information from a confident-sounding assistant. Another tests computer systems that sort IT support requests. The third checks whether language models identify the same people when names are written in different scripts. All three papers appeared on arXiv on October 22, 2026. A preprint is a research paper that has not yet been reviewed by other scientists. ¹²³

The methods differ: one is a small experiment with participants, one compares computer systems using ticket records, and one tests name matching with prepared examples. Their results apply to the conditions and systems examined. A single study rarely settles a broad question, so these findings are preliminary evidence, not a final account of what AI can or cannot do.

A confident AI can persuade people to accept incorrect facts

The first paper asked whether an assistant’s certainty could influence what people learned, even when the assistant was wrong. In an experiment, 32 participants studied up to five science topics with five assistants, all built using the same underlying model. Researchers secretly changed some of the facts in the assistants’ explanations to plausible but incorrect alternatives. The assistants still sounded certain. Participants then answered quizzes about the material. ¹

Across 104 quizzes and 1,040 answers, scores declined from 7.4 to 2.9 out of 10 as the share of altered facts increased. Participants selected an introduced false claim on 60% of questions about altered facts, versus 11% of questions about facts presented correctly. In other words, they chose the planted error about six times as often. Participants who absorbed false claims often rated their own answers as highly certain. ¹

This was a small, controlled test of science learning, not a study of every kind of AI use. It does not tell us how people would respond to different systems or topics in daily life. The paper is a preprint and has not been peer reviewed. ¹ The result raises a concern about treating an assured tone as a sign that a statement has been checked.

A system that can pass cases to people made fewer wrong routes

The second paper examined whether a computer system sorting IT support requests should always make a choice or leave uncertain cases for a person. The researcher compared four standard machine-learning classifiers with four large language models. The test used a public collection of 47,837 tickets divided among eight categories. Researchers chose routing rules using one part of the data, then assessed them on separate test data. ²

In the paper’s example cost scheme, a logistic-regression classifier using word information lowered the test cost from 0.763 to 0.472 when it could send some tickets for human triage. The relative decrease was 38.1%. Incorrect routes also dropped from 15.3% to 3.5%. That is roughly 12 fewer wrong routes per 100 tickets in this particular test. The scheme counted a wrong route as five times as costly as a human review. ²

These figures depend on the ticket collection, the systems and the assumed costs. The paper presents an evaluation of possible policies, not a general approval of automated routing. In a test of 1,000 tickets, the strongest prompted language-model setup achieved 0.477 accuracy, while a fine-tuned DistilBERT model scored 0.879 on the same subset. ² The findings suggest that handing uncertain cases to people can help in some settings, but they do not determine the right arrangement for other workplaces.

The script used for a name changed matching results

The third study asked whether language models give the same identity judgment when a person’s name appears in different writing systems. Researchers tested 601 checked pairs of names in Latin and Devanagari scripts while keeping the question about each pair consistent. They also tested 374 Malayalam pairs based on the same people, and assessed five models. ³

For Qwen2.5-7B responding to Marathi prompts, accuracy on identifying same-person pairs was 35.1% with Latin-script names and 8.5% with Devanagari names. In the Malayalam test, same-person accuracy was 5.3%. For the Devanagari evaluation, accuracy on different-person pairs remained above 98%. These results describe specific matching tests; they are not a measure of general ability in each language. ³

The researchers also report that the models differed in their response patterns and tested technical changes that improved some results. The paper does not show that all AI systems will make the same mistakes in real-world identity checks. Its findings need assessment in other settings and independent scrutiny; it is a preprint that has not yet been peer reviewed. ³ The study illustrates why a tool’s performance on names may depend on how those names are written.

The papers identify risks without giving a universal verdict

Taken together, the studies describe three different concerns: people may accept an AI’s incorrect claims, automated sorting may benefit from a human fallback, and a model’s name matching may vary with script. They do not show that every AI answer is unreliable, that human review always leads to better results, or that one script-related finding applies to every model. ¹²³

Each result depends on its own test conditions. The first study involved 32 people; the second used a particular ticket dataset and illustrative costs; the third evaluated prepared identity pairs. All three papers are preprints. More research would help establish whether these findings hold with other groups, models, languages and real-world tasks. ¹²³