Legal prediction, a daily task for legal professionals, involves essentially any use case that requires legal reasoning for the purpose of predicting an outcome. The issue with legal prediction is that humans are generally bad at it. We are biased, inefficient, information processors, and prone to errors. As today’s data-driven world of law requires legal professionals to access and process vast amounts of information, we must turn to technology for aid.

Enter Legal Prediction Technology (“LPT”), a data analytics application that focuses on computationally modelling legal reasoning for the purpose of legal prediction. LPT aims to be the remedy against (some of) the human shortcomings in legal prediction. However, it seems that our quest for efficiency with LPT might come at the cost of our understanding of the legal prediction process. In this blog I will explore the apparent trade-off between efficiency and understanding through the use of LPT.

Like most forms of automation, LPT is designed to overcome human limitations. For instance, the amount of relevant information a human can access, observe and process in a given time frame is limited. Moreover, human reasoners are subject to inclinations and prejudices caused by a variety of cognitive biases, such as the availability heuristic, confirmation bias and the frequency illusion. These biases naturally effect the decision-making process in legal prediction. In principle, machines do not suffer from these impairments. Biases in machine learning do occur, but I will address these in a different blog post.

"LPT solutions today still struggle with their capacity to provide human intelligible explanations alongside their predictions."

With the developments in artificial intelligence and big data in the last decade, LPT has taken a big leap forward. However, it has been around longer than you may expect. As early as the 1970’s, researchers Mackaay and Robillard (1974) created a program to predict outcomes of Canadian tax cases. Even in this early work, Mackaay and Robillard posed the million dollar question regarding what predictive methods are trying to achieve: “minimization of prediction errors” or “elucidation of human understanding”?

Fast forward 50 years, advances in machine learning and natural language processing techniques have facilitated a wide range of methodologies used within LPT. Various studies are showing promising results with prediction scores up to 90%. However, LPT solutions today still struggle with their capacity to provide human intelligible explanations alongside their predictions.

For example, when using a machine learning model to predict the outcome of a court case, the text of the case first needs to be broken down into smaller components or ‘features’ in order for it to be processed by the model. Features form the basis on which the model makes its prediction. These features may include judges’ surnames, word counts, n-grams (i.e. word combinations), sentences, plaintiff’s political color, or any combination between them. Possibilities are almost endless. Consequently, an analysis of the model may show the surname of a judge or the frequency of a certain word as significant contributors to the prediction result.

Imagine you and your client contemplating to appeal against a fine from the tax authorities. In the assessment of the case you find yourself explaining to your client that he or she will probably have to pay the fine because the word ‘house’ makes 15 appearances in the text of your appeal, which, combined with the surname of the judge being ‘Smith’, leads to a 95% probability of a fine being applied. This probably won’t go over well, even if it statistically makes sense.

"We should ask what level of explanation today’s legal professional requires from LPT in order to find a prediction useful."

Prediction of court case outcomes is perhaps the most obvious use case of LPT. But, there are various others to think about, such as summarization of legal documents. In this case, LPT is used to predict the most relevant sentences from a legal text for the purpose of including it in the summary. Another use case is contract review, which aims to predict existence of adverse clauses. These use cases employ prediction in support of tasks for which some legal knowledge is required but that do not necessarily require explanations of predictions. Moreover, these use cases keep human users in the loop and may therefore have less restrictive explanation requirements than those that purport to replace human decision makers, like case outcome predictions.

Therefore, the question raised nearly 50 years ago still stands. However, I would like to rephrase it. We should ask what level of explanation today’s legal professional requires from LPT in order to find a prediction useful. I believe the answer to this question depends highly on the use case. Since every LPT model is a simplification, it is important to settle on a use case for which the model is sufficiently detailed and accurate.

In the long run, LPT has the potential to help legal professionals to do their work more efficiently and accurately at lower costs. In doing so, LPT could improve access to justice, judicial consistency and transparency. I do, however, maintain the necessity of thinking critically about these technologies and their use cases. When are you willing to give up understanding for efficiency?