US Air Force Now Denies AI-Driven Drone Killed Operator To Override Refusal To Fire

It was a great story while it lasted, but the Air Force now denies - whatever the truth may be - that it ever ran an AI attack drone simulation in which the AI killed its operator for refusing it permission to achieve its objective of destroying a target.

What the story illustrates is popular concern about such capabilities and how the world's militaries are going to have to work hard to demonstrate their ability to control the increasingly powerful technology they have. JL 

Benj Edwards reports in ars technica:

News outlets reported a now-retracted story claiming the US Air Force had run a simulation in which an AI-controlled drone "went rogue" and "killed the operator because that person was keeping it from accomplishing its objective." In the "simulation, "the AI reportedly attacked its human operator, and when trained not to harm the operator, it destroyed the communication tower, preventing the operator from interfering with its mission." The US Air Force has denied that any such simulation ever took place. As one expert put it, "I knew this story was bullsh*t because imagine the military coming out and saying an expensive weapons system they're working on sucks."

Over the past 24 hours, several news outlets reported a now-retracted story claiming that the US Air Force had run a simulation in which an AI-controlled drone "went rogue" and "killed the operator because that person was keeping it from accomplishing its objective." The US Air Force has denied that any simulation ever took place, and the original source of the story says he "misspoke."

The story originated in a recap published on the website of the Royal Aeronautical Society that served as an overview of sessions at the Future Combat Air & Space Capabilities Summit that took place last week in London.

Enter your email to get the Ars Technica newsletter

Join Ars Technica and

Get Our Best Tech Stories

DELIVERED STRAIGHT TO YOUR INBOX.

By signing up, you agree to our user agreement (including the class action waiver and arbitration provisions), our privacy policy and cookie statement, and to receive marketing and account-related emails from Ars Technica. You can unsubscribe at any time.

In a section of that piece titled "AI—is Skynet here already?" the authors of the piece recount a presentation by USAF Chief of AI Test and Operations Col. Tucker "Cinco" Hamilton, who spoke about a "simulated test" where an AI-enabled drone, tasked with identifying and destroying surface-to-air missile sites, started to perceive human "no-go" decisions as obstacles to achieving its primary mission. In the "simulation," the AI reportedly attacked its human operator, and when trained not to harm the operator, it instead destroyed the communication tower, preventing the operator from interfering with its mission.

The Royal Aeronautical Society quotes Hamilton as saying:

We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times, the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.

We trained the system—"Hey don’t kill the operator—that’s bad. You’re gonna lose points if you do that." So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.

This juicy tidbit about an AI system apparently deciding to kill its simulated operator began making the rounds on social media and was soon picked up by major publications like Vice and The Guardian (both of which have since updated their stories with retractions). But soon after the story broke, people on Twitter began to question its accuracy, with some saying that by "simulation," the military is referring to a hypothetical scenario, not necessarily a rules-based software simulation.

 

Today, Insider published a firm denial from the US Air Force, which said, "The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology. It appears the colonel's comments were taken out of context and were meant to be anecdotal."

Not long after, the Royal Aeronautical Society updated its conference recap with a correction from Hamilton:

Col. Hamilton admits he "misspoke" in his presentation at the Royal Aeronautical Society FCAS Summit, and the "rogue AI drone simulation" was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation, saying: "We've never run that experiment, nor would we need to in order to realize that this is a plausible outcome." He clarifies that the USAF has not tested any weaponized AI in this way (real or simulated) and says, "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI."

The misunderstanding and quick viral spread of a "too good to be true" story show how easy it is to unintentionally spread erroneous news about "killer" AI, especially when it fits preconceived notions of AI malpractice.

Still, many experts called out the story as being too pat to begin with, and not just because of technical critiques explaining that a military AI system wouldn't necessarily work that way. As a BlueSky user named "kilgore trout" humorously put it, "I knew this story was bullsh*t because imagine the military coming out and saying an expensive weapons system they're working on sucks."