Timothy Lau* and Alex Biedermann†
Artificial intelligence (“AI”) systems are widely used to assist or automate decision-making. Although there are general metrics for the performance of AI systems, there is, as yet, no well-established gauge to assess the quality of particular AI recommendations or decisions. This presents a serious problem in the emerging use of AI in legal applications because the legal system aims for good performance not only in the aggregate but also in individual cases. This Article presents the concept of using nearest neighbors to assess individual AI output. This nearest neighbor analysis has the benefit of being easy to understand and apply for judges, lawyers, and juries. In addition, it is fundamentally compatible with existing AI methodologies. This Article explains how the concept could be applied for probing AI output in a number of use cases, including civil discovery, risk prediction, and forensic comparison, while also presenting its limitations.
* Federal Judicial Center, Research Division, Thurgood Marshall Federal Judiciary Building, Washington DC. – The views expressed in this Article are of the author alone and do not represent the views of the Federal Judicial Center.
† University of Lausanne, Faculty of Law, Criminal Justice and Public Administration, School of Criminal Justice, 1015 Lausanne-Dorigny (Switzerland). Visiting Researcher at Northwestern University Pritzker School of Law, Chicago IL. – Alex Biedermann gratefully acknowledges the support of the Swiss National Science Foundation through grant BSSGI0_155809.
The authors thank David Kaye of Penn State Law, Edward Imwinkelried of UC Davis, Ronald Allen of Northwestern University Pritzker School of Law, Brandon Garrett of Duke University School of Law, Ian Evett CBE of Principal Forensic Services, and Clare Lau of Johns Hopkins Applied Physics Laboratory for their helpful comments and suggestions.