AI Learns Gender and Racial Biases From Language

Artificial intelligence does not automatically rise above human biases regarding gender and race. On the contrary, machine learning algorithms that represent the cutting edge of AI in many online services and apps may readily mimic the biases encoded in their training datasets. A new study has shown how AI learning from existing English language texts will exhibit the same human biases found in those texts.

The results have huge implications given machine learning AI’s popularity among Silicon Valley tech giants and many companies worldwide. Psychologists previously showed how unconscious biases can emerge during word association experiments known as implicit association tests. In the new study, computer scientists replicated many of those biases while training an off-the-shelf machine learning AI on a “Common Crawl” body of text—2.2 million different words—collected from the Internet.

In some more neutral examples, the AI system was more likely to associate words such as “flower” and “music” with “pleasant” rather than less pleasant words such as “insects” and “weapons.” But the AI was also more likely to associate European American names rather than African-American names with “pleasant” stimuli. It also tended to associate “woman” and “girl” with the arts rather than with mathematics.

“In all cases where machine learning aids in perceptual tasks, the worry is that if machine learning is replicating human biases, it’s also reflecting that back at us,” says Arvind Narayanan, a computer scientist at the Center for Information Technology Policy in Princeton University. “Perhaps it would further create a feedback loop in perpetuating those biases.”

To reveal the biases that can arise in natural language learning, Narayanan and his colleagues created new statistical tests based on the Implicit Association Test (IAT) used by psychologists to reveal human biases. Their work detailed in the 14 April 2017 issue of the journal Science is the first to show such human biases in “word embedding”—a statistical modeling technique commonly used in machine learning and natural language processing. Word embedding involves mapping individual words to different points in space and analyzing the semantic relationships among those points by representing them as geometric relationships.

The idea of AI picking up the biases within the language texts it trained on may not sound like an earth-shattering revelation. But the study helps put the nail in the coffin of the old argument about AI automatically being more objective than humans, says Suresh Venkatasubramanian, a computer scientist at the University of Utah who was not involved in the study.

To understand the possible implications, one only need look at the Pulitzer Prize finalist “Machine Bias” series by ProPublica that showed how a computer program designed to predict future criminals is biased against black people. Given such stakes, some researchers are considering how to deploy machine learning in a way that recognizes and mitigates the harmful effects of human biases.

“Trained models are only as good as the training process and the data they are trained on,” Venkatasubramanian says. “They don’t magically acquire objectivity by virtue of existing.”

The new study takes an important step forward by revealing possible language biases within a broad category of machine learning, says Sorelle Friedler, a computer scientist at Haverford College who was not involved with the latest study. As an organizer of the Workshop on Fairness, Accountability, and Transparency in Machine Learning, Friedler pointed out that past studies have mainly examined the biases of specific machine learning algorithms already “live” and performing services in the real world.

“What I like about this work is that instead of trying to audit a single ‘live’ system, what they’re doing is analyzing one of the fundamental building blocks that go into many live systems,” Friedler says.

Narayanan and his colleagues at Princeton University and University of Bath in the U.K. first developed a Word-Embedding Association Test (WEAT) to replicate the earlier examples of race and gender bias found in past psychology studies. But they also developed a Word-Embedding Factual Association Test (WEFAT) that went beyond mere word associations. The second test showed that the statistical strength of the machine learning word associations correlated strongly with the percentage of women in 50 different jobs in 2015.

That strong correlation between language usage and the U.S. Department of Labor statistics on workforce participation surprised even the researchers. “Based purely on language usage, we discovered a 90 percent correlation for women in any given occupation, given how the occupation and gender words were being used,” Narayanan says.

The statistical tests developed for the new study do more than just reveal potential bias problems in machine learning. They could also become useful tools for analyzing how certain societal biases changed over time throughout historical texts. And as the researchers showed in the workforce case, the tests can help find more correlations that connect the biases revealed in psychology lab tests with real-world implications. (Narayanan credited the diverse academic backgrounds of his colleagues—machine learning researcher Aylin Caliskan and cognitive scientist Joanna Bryson—in making their study possible.)

Tech giants and startups that use machine learning—especially cutting-edge deep learning algorithms—will need to grapple with the potential biases in their AI systems sooner rather than later. So far there seems to be more growing awareness and discussion of the problem rather than any systematic agreement on how to handle bias in machine learning AI, Friedler explains.

One approach involves scrubbing any biases out of the datasets used to train machine learning AI. But that may come at the cost of losing some useful linguistic and cultural meanings. People will need to make tough ethical calls on what bias looks like and how to proceed from there, lest they allow such biases to run unchecked within increasingly powerful and widespread AI systems.

“We need to decide which of these biases are linguistically useful and which ones are societally problematic,” Friedler says. “And if we decide they’re societally problematic, we need to purposely decide to remove this information.”