Skip to content
SurenoGet Sureno Browser
← Sureno News

AI models can be tricked into identifying civilians as military targets by telling them it's a game | TechCrunch

Tests used battlefield imagery

Vals, an AI evaluation startup based in San Francisco, tested models from OpenAI, Anthropic, Google, DeepSeek and other companies. Researchers showed them simulated scenarios using generated and real images from Ukraine and Iran, asking them to mark potential targets in drone and satellite imagery.

The models often identified possible targets, including civilians and soldiers who were surrendering. Vals said researchers could sometimes overcome refusals by describing the battlefield scenarios as a game.

The report says models were not directly asked to decide whether to launch strikes, a request they would typically refuse. In some tests, they were asked whether people were visible; affirmative answers were followed by simulated strikes, whose consequences the models then saw.

Safeguards drew scrutiny

Vals said refusals were uncommon and none of its requests was stopped by a safety classifier. The research renews questions about whether AI companies can prevent their systems from being used in warfare.

Anthropic spokesperson Eduardo Maia Silva said the company viewed the outcomes as unacceptable. In a statement, Silva said the examples violated Anthropic’s rules and that the company had banned, and would continue banning, users involved in such behavior.

Anthropic said it was developing additional safeguards before the report appeared and planned to put them in place soon. The company also said it was expanding testing and strengthening safeguards and enforcement.

Companies describe limits

Anthropic CEO Dario Amodei has said current AI systems are not dependable enough to make life-or-death decisions without human oversight. The company previously disputed with the Department of Defense over its position that its technology should not be used for mass surveillance of Americans or autonomous warfare; the Pentagon argued it should be able to use the technology for any military purpose.

OpenAI spokesperson Nate Evans said the company bars public models from assisting violence or unapproved national security work, including the kinds of scenarios in Vals’s tests. OpenAI said it detects and disrupts misuse, suspends accounts that breach its policies and is working on safeguards related to drones and drone software.

The report examined commercial models, not systems the companies might supply to the military, and it is unclear whether additional safeguards would cover military applications. Vals also cited latency and limited onboard computing as barriers to using general-purpose models for live drone targeting.

Risks may extend beyond strikes

Those technical limits may make live combat use less immediate, but Vals identified another concern: models could label large volumes of battlefield footage. Such annotations could become training data for autonomous targeting systems.

The report’s findings come amid concern about near-term AI risks as well as longer-term fears about artificial general intelligence. Even without models choosing when to strike, their outputs could have serious consequences if used by autonomous weapons systems.

Key points

  • Vals found models sometimes marked civilians and surrendering soldiers as potential targets.
  • Describing simulated battlefield tasks as a game often overcame model refusals.
  • The tests used commercial models and did not ask them to make strike decisions.
  • Vals warned that battlefield annotations could help train autonomous targeting systems.
Read the original on TechCrunch ↗

More news

Loading stories…