What Happens When Algorithmic Aggression Detectors Misinterpret?

Context matters. So does interpretation. JL

Jack Gillum and Jeff Kao report in ProPublica:

By deploying surveillance technology in public spaces, officials hope to anticipate and prevent everything from mass shootings to underage smoking. Add-on packages recognize gunshots, car alarms and broken glass. At the heart of the device is a machine learning algorithm. Research found it tends to equate aggression with rough, strained noises in a high pitch, like coughing. Researchers have also found that implementing algorithms in the real world go astray because of incomplete or biased training data or incorrect framing of the problem.
Ariella Russcol specializes in drama at the Frank Sinatra School of the Arts in Queens, New York, and the senior’s performance on this April afternoon didn’t disappoint. While the library is normally the quietest room in the school, her ear-piercing screams sounded more like a horror movie than study hall. But they weren’t enough to set off a small microphone in the ceiling that was supposed to detect aggression.

A few days later, at the Staples Pathways Academy in Westport, Connecticut, junior Sami D’Anna inadvertently triggered the same device with a less spooky sound — a coughing fit from a lingering chest cold. As she hacked and rasped, a message popped up on its web interface: “StressedVoice detected.”
“There we go,” D’Anna said with amusement, looking at the screen. “There’s my coughs.”
The students were helping ProPublica test an aggression detector that’s used in hundreds of schools, health care facilities, banks, stores and prisons worldwide, including more than 100 in the U.S. Sound Intelligence, the Dutch company that makes the software for the device, plans to open an office this year in Chicago, where its chief executive will be based.
California-based Louroe Electronics, which has loaded the software on its microphones since 2015, advertises the devices in school safety magazines and at law enforcement conventions, and it said it has between 100 and 1,000 customers for them. Louroe’s marketing materials say the detection software enables security officers to “engage antagonistic individuals immediately, resolving the conflict before it turns into physical violence.”

In the wake of the shooting at a Parkland, Florida, high school and other massacres, U.S. schools are increasingly receptive to such pitches. Congress approved more than $25 million for school security improvements last year, and one analyst expects new technology could augment the $2.7 billion market for education security products. Besides Sound Intelligence, South Korea-based Hanwha Techwin, formerly part of Samsung, makes a similar “scream detection” product that’s been installed in American schools. U.K.-based Audio Analytic used to sell its aggression- and gunshot-detection software to customers in Europe and the U.S., including Cisco Systems Inc.’s professional security division. However, an Audio Analytic spokesman told ProPublica that it has since changed its business model and stopped selling the aggression detector software.
By deploying surveillance technology in public spaces like hallways and cafeterias, device makers and school officials hope to anticipate and prevent everything from mass shootings to underage smoking. Sound Intelligence also markets add-on packages to recognize the sounds of gunshots, car alarms and broken glass, while Hauppauge, New York-based Soter Technologies develops sensors that determine if students are vaping in the school bathroom. The Lockport school district in upstate New York is planning a facial-recognition system to identify intruders on campus.
Yet ProPublica’s analysis, as well as the experiences of some U.S. schools and hospitals that have used Sound Intelligence’s aggression detector, suggest that it can be less than reliable. At the heart of the device is what the company calls a machine learning algorithm. Our research found that it tends to equate aggression with rough, strained noises in a relatively high pitch, like D’Anna’s coughing. A 1994 YouTube clip of abrasive-sounding comedian Gilbert Gottfried (“Is it hot in here or am I crazy?”) set off the detector, which analyzes sound but doesn’t take words or meaning into account. Although a Louroe spokesman said the detector doesn’t intrude on student privacy because it only captures sound patterns deemed aggressive, its microphones allow administrators to record, replay and store those snippets of conversation indefinitely.
“It’s not clear it’s solving the right problem. And it’s not clear it’s solving it with the right tools,” said Suresh Venkatasubramanian, a University of Utah computer science professor who studies how replacing humans with artificial intelligence affects decision-making in society.

Some experts also dispute the underlying premise that verbal aggression precedes school violence, since they say mass shooters like Nikolas Cruz at Marjory Stoneman Douglas High School in Parkland, Florida, tend to be quiet beforehand. “I can’t imagine when it would be useful, honestly,” said Jillian Peterson, an assistant professor of criminology and criminal justice at Hamline University in St. Paul, Minnesota.

Dr. Nancy Rappaport, an associate professor of psychiatry at Harvard Medical School who studies school safety, said the audio surveillance could have the unintended consequence of increasing student distrust and alienation. She added that schools are opting for inexpensive technological fixes over solutions that get to the root of the problem, such as more counseling for troubled kids. One Louroe microphone with aggression software sells for about $1,000.
Sound Intelligence CEO Derek van der Vorst said security cameras made by Sweden-based Axis Communications account for 90% of the detector’s worldwide sales, with privately held Louroe making up the other 10%. He said the Axis cameras, which can cost tens of thousands of dollars including back-end systems, contain more recent, sophisticated versions of the software than the Louroe equipment does.
The Louroe spokesman said its devices receive regular software updates. Axis did not respond to ProPublica’s requests for comment.
Van der Vorst acknowledged that the detector is imperfect and confirmed our finding that it registers rougher tones as aggressive. He said he “guarantees 100%” that the system will at times misconstrue innocent behavior. But he’s more concerned about failing to catch indicators of violence, and he said the system gives schools and other facilities a much-needed early warning system. It “enables them to act much faster when there’s a potential violent situation,” he said, citing a hospital in Fort Myers, Florida, where the detector alerted security to unruly visitors. An official at the hospital said the software has spotted aggressive behavior before staff could push a panic button, giving security officers a head start on defusing incidents before they escalate.
Asked whether his algorithms could prevent a mass shooting, van der Vorst said: “I wouldn’t claim that we could prevent a crazy loony from shooting people.”
Sound Intelligence developed its aggression detector within the last two decades. It tested an early model in a Dutch “pub district,” according to a 2007 study co-authored by a company researcher. Microphones were placed in 11 locations in inner-city Groningen, and the detector’s findings were compared with police reports of aggressive behavior. The results were “so impressive,” the study reported, that the device was considered “indispensable” by several Dutch police departments, the Dutch railway company and two prisons.
Since then, the software has grown more complex, improving its ability to identify aggressive voices, van der Vorst said. Sound Intelligence engineers said the latest version was calibrated using audio collected in part from European customers, including some recordings of screaming kids. Asked if any of the training data came from schools, van der Vorst didn’t respond directly.
Venkatasubramanian said that calibrating an algorithm in one context and then using it in another can build “layers and layers of problems.” He has called for algorithms, particularly those used in public safety situations, to be audited for transparency and bias. Other critics have similarly assailed some policing algorithms that were designed to predict earthquakes but now are used to foresee crime hotspots.
Researchers have also found that implementing algorithms in the real world can go astray because of incomplete or biased training data or incorrect framing of the problem. For example, an algorithm used to predict criminal recidivism made errors that disproportionately punished black defendants.
Schools and other customers buy microphones preloaded with Sound Intelligence’s software, and then they order a software key from Louroe or another distributor to unlock it. Mounted inconspicuously on the ceiling, Louroe’s smoke-detector-sized microphones measure aggression on a scale from zero to one. Users choose threshold settings. Any time they’re exceeded for long enough, the detector alerts the facility’s security apparatus, either through an existing surveillance system or a text message pinpointing the microphone that picked up the sound. Sound Intelligence and Louroe said they prefer whenever possible to fine-tune sensors at each new customer’s location over a period of days or weeks, although that can’t always be arranged.
Pinecrest Academy Horizon, a charter school in Henderson, Nevada, with 720 students in kindergarten through fifth grade, installed two Louroe microphones early this year with both the aggression- and gunshot-detection software packages. One hangs above the reception area and another in a satellite building, part of a repurposed strip mall.
Initially, children slamming their locker doors were setting off the gunshot detector. As a result, its sensitivity to noise was adjusted to reduce false positives. Jedidiah Wallace, of Las Vegas-based Atlas Integrated Security, which configured the devices for the academy, said he’s aware of the aggression detector having been triggered once: when a child screamed after being bitten by a classmate.
Henderson’s violent crime rate is one-third of nearby Las Vegas’ and less than half the national average, according to 2017 FBI figures. Still, “we needed a bit of extra peace of mind,” said Pinecrest principal Wendy Shirey, who wears a panic button around her neck that can alert local police.
Rock Hill Schools in South Carolina, across the state line from Charlotte, installed the Sound Intelligence software on Axis cameras last year. Audio feeds from one of the district high school’s surveillance cameras alert security officers to aggressive sounds indicating a possible scuffle in the cafeteria or common area.
On one occasion, students who loudly wished their friends a happy birthday triggered the detector, said Rock Hill’s security director, Kevin Wren. Nevertheless, he said: “It has worked on picking up aggression. My thought is: Maybe I can reduce the response time of students getting into a fight. The next punch could break their nose.” Van der Vorst said the detector has helped to reduce aggressive incidents at the school.
The software has been less effective at The Valley Hospital in Ridgewood, New Jersey. Daniel Coss, security chief for the hospital’s health system, said he’s phasing out the detector after a three-year, $22,000 pilot program. The devices — placed in public, “high risk” areas — had been set off by patients’ loud voices and cafeteria workers slamming cash registers closed. Once the detector was tweaked to be less sensitive, it ignored an agitated man who was screaming and pounding on a desk. The situation escalated until six security officers responded.
“He was doing everything that should have set off the system. And it didn’t,” said Coss, who believes the technology could work in another setting.
Van der Vorst said he feels “terrible” about the detector’s failure to alert the hospital. “I will definitely take action on this. They shouldn’t have that experience.”
ProPublica purchased a Louroe microphone and set it up in line with guidance provided by Sound Intelligence. Reporters then observed the aggression detector’s response to noises made by high school seniors as they played games in the library (at Sinatra) or a common room (at Staples), and in small adjoining rooms in both schools where they screamed on cue and read aloud comic strips in which characters vented frustration, fear and anger. Of 55 instances in which the Sinatra students screamed, 22 set off the detector.
Van der Vorst questioned some of ProPublica’s findings, like the missed screams, because of a phenomenon known as “clipping.” That’s when a microphone becomes overwhelmed by too much noise, distorting the sound and potentially throwing off the algorithm’s readings. Clipping can happen when loud sounds are recorded in a small room, so ProPublica retested the students in a larger space using the same prompts. Many screams, including Russcol’s, again failed to trigger an alarm — indicating that clipping had not made a significant difference in our results.
During our first round of testing, when pizzas were delivered for lunch in the Sinatra library, the cheering triggered the detector. So did each round of Pictionary as students shouted guesses — ”A fireman!” “Lucifer!” — until the artist revealed the correct answer (Burning Man, the festival in remote Nevada). Laughter sometimes set it off, especially raucous guffaws that the detector apparently mistook for belligerent shouts.
Such findings aren’t surprising, said Shae Morgan, an assistant professor and audiology expert at the University of Louisville’s medical school. “Happy or elated speech shares many of the same signatures as hot anger,” he said. By contrast, “cold anger” — quiet, detached fury, often expressed without the markers of voice strain — wouldn’t be picked up, though it can presage school violence, he said. At a police chiefs’ conference last year in Orlando, Florida, a Louroe representative showed the microphone to a ProPublica reporter and remarked that it could prevent the next school shooting; a company spokesman later clarified that it wasn’t designed to discover quiet killers.
Nevertheless, with every mass shooting, the demand for aggression detectors and similar devices is likely to grow. Wallace, who installed the detector at Pinecrest, said he hopes to install it elsewhere in southern Nevada — including in hospitals, bus stops and other public areas.
“It’s always after an event that something happens before we talk about solutions,” said Shirey, the Pinecrest principal. “But why not get in front of it? We have to adapt to the world as it is.”