The unprecedented interest, investment, and deployment of machine learning across many aspects of our lives in the past decade has come with a cost. Although there has been some movement towards moderating machine learning where it has been genuinely harmful, it’s becoming increasingly clear that existing approaches suffer significant shortcomings. Nevertheless, there still exist new directions that hold potential for meaningfully addressing the harms of machine learning. In particular, new approaches to licensing the code and models that underlie these systems have the potential to create a meaningful impact on how they affect our world. This is Part I of a two-part essay.
The use and misuse of machine learning systems
Before discussing moderation, it is worth first asking if machine learning merits additional consideration about its usage when compared to other forms of software. In order to answer that question, we first need to examine how machine learning is being used, and in particular, the ways that systems built with machine learning as critical components are being misused.
Law enforcement is one area in particular that has shifted to use, and in some cases, rely on machine learning technology to inform both overall policy and individual investigations, in the forms of predictive policing algorithms and facial recognition. In 2019, Robert Williams, a Black man, was falsely arrested in Detroit for shoplifting based on faulty facial recognition, making national news. Then later that year, the same thing happened again to another Black man, named Michael Oliver. This instance was, in part, driven by fundamental weaknesses of using facial recognition. In this case, the system was facial recognition, which completely missed the fact that the man seen in the video used to “identify” Michael didn’t have a visible tattoo on his bare arms, as opposed to Michael’s heavily inked ones. It’s not just Detroit where this occurs-it’s everywhere. The same year, a Black man named Nijeer Parks in New Jersey was falsely imprisoned for 10 days with the only support for his arrest being evidence provided by a facial recognition system.
It shouldn’t come as a surprise that many of the people who are being arrested using these systems are Black. Machine learning systems have a well-documented bias in practice regarding facial recognition, with multiple studies repeatedly demonstrating an inability to produce equal accuracy in practice on darker faces and those belonging to women. This performance gap is well known, yet even the ability for third parties to regularly audit the models for these biases that can accidentally lead to unnecessary police action against law-abiding citizens, arguably the bare minimum for deploying them to production, is startlingly rare.
Questionable conduct made possible by machine learning doesn’t just happen by accident, either; during the Black Lives Matter protests in late summer of 2020, the NYPD used facial recognition to identify activist Derrick Ingram, sending a large number of officers to his home. And it’s not just the police that are using these tactics; counterprotesters are also leveraging facial recognition through open source tools to identify the officers participating in the responses to these protests, specifically naming open source libraries like Tensorflow as critical to this effort. However, none of this stops at the level of individuals. Compare a single targeted use of facial recognition to the kind of system being used in Pasco County, FL, which uses pseudo-scientific methodologies to systematically target minorities and the underage [1].
In a way, though, these tactics are just coming home. Just as the Pasco County Sheriff's Office has a number of ex-military intelligence personnel working to drive these kind of programs, persistent surveillance systems of incredible scale have been implemented in warzones across the Middle East by companies like Palantir, which also have run point on building out and deploying predictive policing programs utilizing cutting-edge data science in municipalities like the one in 2018 in New Orleans without civilian oversight even by the city council, making American civilians unwary test subjects in programs with roots set in identifying terrorists planting IEDs (note: Palantir has since ended this program). This kind of persistent, pervasive system is also being taken to the next level by companies like ClearviewAI, which collects, sorts, and offers facial recognition using online photos, eliminating even the geographical barriers on data collection found in other approaches.
The key takeaway here is this: the abuse of machine learning tools is serious, widespread, and systemic, not simply isolated incidents in systems that otherwise work. Even compared with other types of software (operating systems, desktop GUIs, web browsers), machine learning demonstrates a unique capacity to be systematically used in ways that can and are being used to harm individuals, vulnerable groups, and society as a whole.
Existing approaches towards the ethical use of machine learning
In some situations, market action might resolve the problem without legal or regulatory intervention–for example, the one-year moratorium that Amazon Rekognition placed on police use of their product starting in June 2020. This is, oddly enough, one of the benefits of keeping certain code under a proprietary umbrella, whether through platform access or a proprietary software license that places limitations on who is allowed to use it and how.
In other situations, those closest to the problem are often the ones who are most interested in developing solutions. In response to growing concern about the methods and applications of their tools, organizations like the Association for Computing Machinery (ACM), one of the largest computing professional groups at nearly 100,000 members, has created a Code of Ethics for its members. This includes at least one section that states “Extraordinary care should be taken to identify and mitigate potential risks in machine learning systems.” And they’re not alone on this. Other groups in academia have produced documents like the Montreal Declaration, from the Université de Montréal, which aims to build an ethical framework and open a dialogue around the responsible development and deployment of AI development. And corporations like Google, Microsoft, and even more traditional engineering companies like Rolls-Royce are putting together ideas about how to set a standard for responsible use and trust with these technologies.
Finally, government regulation has also been a key response in attempting to reduce the harmful ways in which machine learning can be used. In the United States alone, major metropolitan areas like Boston, Portland (both in Maine and Oregon), and Minneapolis have all placed major limits on facial recognition use in public areas and by government officials including police. Other areas have taken a slightly different approach. Since machine learning techniques generally benefit from increased amounts of data, some restrictions around what data can be harvested from the public have been created, with the teeth to potentially make a difference. Notably, Facebook was fined $650 million in 2020 for violating Illinois’ restrictions on using biometric data without consent. At a higher level, the European Union’s General Data Protection Regulation (GDPR), implemented in mid-2018, has forced a reckoning across the online world about how the data often used to feed these machine learning systems is collected.
Yet the underlying weakness with most of these approaches is that adherence to best practices doesn’t occur voluntarily. The truth is that a significant subset of the groups looking to use machine learning are not good-faith actors when it comes to using these techniques responsibly. The cost of a highly-vetted machine learning model with practices like regular regression testing, adversarial mitigation, and analysis of training data bias, is fundamentally more expensive than a naive version of the same model that might offer surface-level improvements over existing methods. A fully-developed facial recognition model with uniform performance across sex and race is more expensive than one that does well on white males, and building a natural language interpreter without an anti-Semitic bias is more difficult and expensive than one trained on unfiltered Internet data.
However, because of the lack of effective oversight on how those models are used, there is little incentive on the end user side to pay those costs, and therefore little incentive for the vendors of the technology to unilaterally invest in the specialists or tooling required to meet that standard. Instead, best practices actually represent a cost center with low return, which means that natural market mechanisms are exceedingly unlikely to drive improvements in these areas.
Relying on the market to provide an effective path for implementing ethical considerations has fallen short in several clear incidents. While on one hand, Rekognition might be willing to place a moratorium on police use of its products until the effects can be more thoroughly examined, researchers at corporations like Huawei have not shied away from producing models specifically aimed at identifying ethnic minorities despite the clear and present danger of abuse, especially in light of the ongoing human rights abuses against the Uighur minority by the Chinese government.
The reality, however, is that even when one group refuses to work on a technology on ethical grounds, another group is almost always willing to step into that space to pick up the work instead. Consider Project Maven, a Department of Defense project aimed at using machine learning to improve the accuracy of object detection in warzones, likely for drone strikes. Researchers at Google, which originally picked up the contract, created so much pushback on this work that the company eventually dropped the program. However, the work was quickly picked back up by another corporation, Anduril Technologies. This demonstrates one of the key flaws of relying on market solutions as a way to regulate potential ethical concerns of a technology; moral relativism shows up very quickly, because even if everyone acknowledged an independent responsibility to act ethically, different groups have very different definitions of what that means.
It’s certainly possible to argue that this technology as a whole is relatively young, and therefore gauging the ability of market structures to point towards ethical decision-making requires guesswork. And to a degree, those arguments have some merit. AlexNet was only released in 2012. Deepfakes didn’t really exist until 2017. However, as previously mentioned, machine learning and data science systems are not solely composed of the algorithms needed to process the data, but also the components used to collect that data in the first place. It’s not a far stretch to imagine that the same standards of conduct applied to data collection will end up being applied to the ways that actionable results are derived from them as well. And in those mature systems, it’s easy to see the degree to which future systems could become additionally intrusive and exploitative, because modern data collection practices can be a digital hellscape filled with incentives for identifying and profiling pregnant women and children and silently building trackers into the tools used to create websites, even when those websites might service enormously vulnerable groups like transgender military service members and veterans. If this is the status quo that has been allowed, and perhaps even encouraged, to proliferate one end of the data pipeline, it’s difficult to imagine that the other end has the potential to look much different in the future, especially in light of the glimpses that have already been seen.
Of course, a number of organizations, whether in academia, single corporations, or larger coalitions of practitioners have tried to encourage their members to act responsibly with machine learning. In some fields like medicine or law, this approach could prove very important. However, software developers are not generally licensed, and professional associations are entirely voluntary. Comparatively, the first law pertaining to engineering licensure in the United States was passed in Wyoming in 1907, largely due to poor submission quality for public project proposals from groups or individuals lacking professional training. Licensing requirements for the developers of government-purchased machine learning services could help fill this gap. In the meantime, however, the lack of formal licensing requirements does have a clear effect: compulsory discipline for unethical conduct or ignoring best practices is impossible in the way that is possible in other professions by debarring a lawyer for lying to the court or relieving a doctor of their license for malpractice.
Conversely, the laws we’ve examined above do offer a way to regulate machine learning in the public interest even when those groups don’t want it. Unfortunately, there’s a growing body of evidence that’s pointing a direction showing that even when laws prohibit the use of machine learning in certain situations, actors both inside and outside the government either don’t obey those laws, or deliberately obscure their use of these techniques in the first place. For example, the Los Angeles Times reported in late 2020 that since 2009, LAPD officers have run over 30,000 facial recognition scans, despite responding to a 2016 inquiry from the Georgetown University Center on Privacy and Technology inquiry on the department’s use that it had “no records responsive to request” (the LAPD has since created a commission to review facial recognition use with the department, and set up additional limits on use). More recently, BuzzFeedNews reported in April 2021 that the NYPD had repeatedly denied having any relationship with ClearviewAI, despite clear evidence to the contrary revealed through a set of public records requests through the New York Freedom of Information Law.
Furthermore, The Markup has reported on persistent instances in at least 5 cities across the country where police departments have deliberately sidestepped bans or used loopholes to access machine learning-built facial recognition by relying on either external vendors or using information given to them from sister agencies with roots originating in the technology [2]. This kind of two-step isn’t unique to facial recognition technology either; it’s part of a broader trend in agencies using loopholes to obtain the information they want via proxy, even in situations where the spirit of the law is fairly clear.
What's an individual to do?
So if a concerned party, such as a machine learning developer or researcher, wanted to try to effectively attempt to limit the amount of harm their work can do to other people, what could they do? They could lobby their government for some sort of prohibition--but, as has been shown, the complexity of these issues and the pushback from other groups with a vested interest in preventing that regulation will likely limit the effectiveness of this approach for some time to come. They could yell at someone on Twitter--after all, public outrage has occasionally led to reconsideration of an organization’s stance (though this is a rare event). They could keep their code behind a paywall and/or release their code under a proprietary license that allows them to examine and approve each situation in which a third-party wants to use their code. Or, they could just stop developing machine learning code at all. That last one might seem glib, but it’s a path that a number of machine learning practitioners, including some very talented ones, have chosen to take.
An example of this is Joseph Redmon, the lead developer of the Darknet neural network library and original author of the state-of-the-art You Only Look Once (YOLO) object detection algorithms, which have cumulatively garnered over 27,000 academic citations. In the conclusion section of the YOLOv3 paper, he wrote:
I have a lot of hope that most of the people using computer vision are just doing happy, good stuff with it, like counting the number of zebras in a national park, or tracking their cat as it wanders around their house. But computer vision is already being put to questionable use and as researchers we have a responsibility to at least consider the harm our work might be doing and think of ways to mitigate it. We owe the world that much.
In fact, Redmon no longer works in the computer vision field due to being uncomfortable with being the way that his research was being funded and used. As Joe has pointed out, in the same way that others have (including the RustConf 2020 keynote address), technology is not neutral. A person’s decision to engage with or contribute to a technology is inherently a judgement on the downstream effects on that decision. For those machine learning researchers and developers who feel the costs of their work outweigh the benefits, this may seem at the moment like one of the only paths forward.
There is, however, a key assumption built into this discussion: the assumption that machine learning programs developed are freely available to use by the public in any way they see fit. If we were talking about almost any other form of intellectual property, this discussion would look very different. After all, it would be possible, and indeed expected, that valuable new technological advances, or more generally any form of content, would be placed under the creator’s ownership and control through copyright and/or the patent system.
Part II of this essay is forthcoming next week.
Author Bio: Chris is an open source developer and contributor to the Rust-ML machine learning working group. He holds a B.S. in Mechanical Engineering and Physics from Northeastern University.
Note: The author has received partial funding from the U.S. Department of Defense for unrelated work.
Citation
For attribution in academic contexts or books, please cite this work as
Christopher Moran, "Machine Learning, Ethics, and Open Source Licensing (Part I/II)", The Gradient, 2021.
BibTeX citation:
@article{moranopensource2021,
author = {Moran, Christopher},
title = {Machine Learning, Ethics, and Open Source Licensing (Part I/II)},
journal = {The Gradient},
year = {2021},
howpublished = {thegradient.pub/machine-learning-ethics-and-open-source-licensing/},
}
){kind=link}
