This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.
In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.
I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.
AI deception: How to define it?
If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.
What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”
Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.
Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways.
What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.
Preparing ourselves against AI deception
The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks.
This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot.
Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors.
We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.
However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.
Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.
References
1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.
2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982.
3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.
4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.
5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.
Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.
