Autonomous agents built on large-language models are entering organizational workflows as quasi-employees that browse, code, and decide without continuous human supervision. This article introduces AI agents as objects of management control by conducting several experiments and reviewing the literature on AI safety research. The first two experiments show that setting targets for an AI agent can lead to misreporting and cheating. The third experiment indicates that AI agent’s budget forecasts become downward biased if the agent is informed that a too high forecast leads to project termination. Subsequently, the paper provides examples from AI safety research that indicate that the control problems apparent in the three experiments are part of a broader issue. AI agents have been shown to distort effort, manipulate data, and bypass monitoring. The article continues by identifying four sources of misaligned AI agent behavior: (1) training of the AI models, (2) misspecified performance metrics, (3) goal conflicts between prompts at different hierarchical levels such as system prompts and user prompts, and (4) the lack of separation between data and instructions. Next, the article discusses potential solutions to AI agent control problems through the classic three-legged stool of decision rights, monitoring and measurement, and incentives. The article concludes with suggestions for future research.

Professor Matthias Mahlendorf

Frankfurt School of Finance and Management | Google Scholar profile

Bio: Matthias D. Mahlendorf is a Professor of Managerial Accounting at Frankfurt School of Finance and Management. He is interested in the areas of performance measurement – especially with respect to managing sustainability performance – as well as the digital transformation of management control. He is the Editor-in-Chief of the journal Management Accounting Research. He has published in journals such as Journal of Accounting Research, Management Science, and The Accounting Review. He is a member of the editorial board of The Accounting Review and serves as a reviewer for Journal of Accounting Research, Review of Accounting Studies, Production and Operations Management, and other journals.

Source: Adapted from https://www.frankfurt-school.de/en/home/research/staff/Matthias-Mahlendorf