Much of the public awareness that criminal justice decisions are being influenced by risk assessment tools started with an instrument called the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS). A 2016 investigation by ProPublica found that the artificial intelligence-based tool, which several U.S. jurisdictions use to determine a defendant's or convicted offender's risk to reoffend and factors contributing to that risk, was biased against African Americans.

The company that owns the tool, Equivant (formerly Northpointe), disputed the finding, citing research from independent academic studies that showed otherwise. What followed was a surge of ongoing research and debate over the impartiality, effectiveness and limits of COMPAS and other risk and needs assessment (RNA) tools, and the role they should play in the criminal justice system.

It's not hard to see why COMPAS, which has influenced sentencing and offender supervision decisions around the country, has sparked so much attention. It piques the imagination: an AI system predicting future crimes, like something straight out of "Minority Report," "1984″ or other dystopian sci-fi tales.

But the reality of risk assessment tools is far less clear-cut, or dramatic. COMPAS is just one of many RNA instruments U.S. jurisdictions use to assess a person's risk of recidivism and the factors contributing to that risk (i.e., criminogenic needs). None of these tools, including COMPAS, are prescient. And while COMPAS uses AI technology (specifically machine learning-based algorithms), it is the rare exception rather than the rule. Most, if not all, other RNA tools are considerably less advanced and far more transparent and malleable.

And to be sure, the scores generated from these tools are only one of many factors judges and criminal justice officials take into account when deciding a case, or determining the level of supervision and programming an offender receives.

Of course, this doesn't mean the debate around risk assessment tools isn't necessary. But the debate cannot progress without an understanding of the specific assessment instruments deployed across the U.S., and how they're used, created and validated.

As the culmination of a yearlong research project, Legaltech News is planning to closely examine these differences and how they shape the reality of risk assessment in the U.S.

Over the next five days, we'll discuss how risk assessment tools are being chosen, implemented and quality tested, which can often influence how much—or how little—of an impact they have on the lives of those who enter local courts and prisons. We'll also look at how these tools work, how their scores are created, and why some argue they help reduce bias within the criminal justice system, while others say they just perpetuate and exacerbate the problem.

|

The Research Project

In spring 2019, Legaltech News decided to embark on a research project to undercover what risk assessment tools are in use across all 50 states, and how these states are deploying them.

It was a fairly large undertaking. As it turns out, there are a multitude of different tools in use not only across individual states, but in individual counties and cities. Some states mandate risk and needs (or just solely risk) assessments in parts of their criminal justice systems, though not all required localities to use a specific tool. In fact, some let local jurisdictions decide what tool works best for their needs. In other cases, counties took the lead in implementing, and even creating, a tool of their choice, absent any direction from their state government. Counties in Arizona and Minnesota, for instance, built their own risk assessment tools, which were eventually adopted statewide.

In addition to RNA tools for general adult populations, we also found a variety of specialized tools in use, such as those for juvenile offenders or ones that assess a defendant's or convicted offender's risk of committing domestic or sexual violence.

A 2013 academic study of risk assessment instruments in the U.S. found over 60 commercial and state-developed tools for the general adult population. David D'Amora, senior policy adviser at the Council of State Governments Justice Center, notes that when you add in specialized tools, "you're actually looking at another 100. … There is actually a fairly large number of [risk assessment] tools that are in use in the U.S. and have been validated in the U.S."

For courts, probation departments, parole boards, pretrial agencies, and others, there's no lack of options. "Almost everybody that is going to use an assessment tool has an assessment tool, and when they go to make a change, they either look at what's out there or they decide to do something themselves. So from an economic perspective the market is pretty saturated," says Dr. Edward Latessa, a professor and director of the School of Criminal Justice at University of Cincinnati who helped develop the Ohio Risk Assessment System (ORAS), a set of 10 different assessment tools.

As we conducted our initial research, however, we saw a trend emerge: despite the sheer number of tools on the market, there were certain names that were popping up repeatedly. This was no coincidence. D'Amora notes that despite the diversity of risk assessment instruments, "there are certain tools that are out there that are the best known and used in a lot of places."

With this in mind, we refocused our project, exclusively looking at tools used on the largest general adult population in each state. This included both well-known instruments offered by for-profit technology developers, public domain platforms created and managed by universities, and proprietary tools developed by individual states themselves.

While most of the instruments we found were RNA tools that measure a person's risk of recidivism and criminogenic needs, some were exclusively pretrial risk tools that only measure the risk of a defendant failing to appear in court and/or committing a crime during the pretrial period. Unlike RNA tools, which measure both risk and needs and can be used at multiple points of the criminal justice system, pretrial risk tools only measure a particular type of risk.

We included the pretrial tools in our project because of their prevalence—in many states, they were the most widely used tool. COMPAS and ORAS also include pretrial risk assessment tools as part of their broader multiproduct offering. What's more, pretrial hearings represent an important part of the criminal justice system, and use of pretrial risk instruments is becoming more widespread as bail reforms pick up speed across the U.S.

To be sure, our research is a snapshot of RNA and pretrial risk tools across the country at the beginning of 2020. It is possible that there have been some changes since then, in part due to the COVID-19 pandemic and subsequent economic recession. At the time we conducted our research, states such as Michigan and New Hampshire were piloting specific assessment tools to determine if they could roll them out more broadly.

With state funding efforts facing a different reality than they had at the beginning of 2020, it's likely that some of these efforts were impacted.

|

What We Found

How and to what ends jurisdictions across the U.S. use risk assessment tools, and at what decision points in the criminal justice system they deploy them, varied considerably across the country. However, most of what we found were RNA tools that criminal justice officials used to determine what type of programming and supervision convicted offenders—whether in prison, or on probation or parole—should receive to help lower their risk of recidivism.

Some states, such as Arizona, Florida, Idaho, North Dakota and Ohio, among others, periodically reassess offenders with RNA tools as they make their way through one or more steps in the criminal justice system, such as from presentence to prison programming and finally parole. This is done to determine whether the programming and supervision an offender received was effective in lowering their risk score.

Offenders can be reassessed periodically because RNA tools look not only at static factors, such as age at first conviction and criminal history, but also dynamic factors such as substance use, education, and employment, which can change over time.

Some RNA offerings, such as the ORAS, are designed specifically with reassessment in mind, while others such as the Level of Service/Case Management Inventory (LS/CMI) integrate into a case management technology to help agencies organize and host all information related to an offender's supervision.

Almost all the static and dynamic information assessment tools required to generate their scores is obtained via in-person interviews and/or criminal justice records. Unlike RNA tools, pretrial risk instruments only use static information because they only focus on one particular point in the criminal justice process.

Some jurisdictions around the U.S. will also use supplemental assessment tools in conjunction with their general RNA tool. Some supplemental tools act as "screeners" to determine whether an offender is high risk enough to assess with a more comprehensive RNA instrument.

Oregon and Hawaii, for instance, first assess offenders with state-built assessment instruments to help determine what probation or parole programming they receive. If offenders score high enough, Oregon and Hawaii will then screen them with the more in-depth Level of Service Inventory-Revised (LSI-R) or the LS/CMI, respectively, to get a deeper understanding of their criminogenic needs.

There is also one supplemental assessment tool that is gender-specific: The Women's Risk and Needs Assessment (WRNA) that, similar to ORAS, was created by the University of Cincinnati. The WRNA is used on female prison populations in at least two states, Kansas and Montana, based on the idea that the risk factors contributing to someone reoffending are different for women and men (we'll dive deeper into that topic in an upcoming article).

Criminal justice systems around the U.S. not only differ in how broadly or narrowly they use risk assessment tools, but how regularly they deploy them as well. In some states, such as New York and Colorado, the use of certain risk assessments is optional and up to the discretion of court and correctional officials. In other states, such as Georgia and Ohio, it's required at certain decision points under state law.

The sophistication of tools used across the states also varies considerably. On one end of the spectrum are state-built tools such as the Colorado Pretrial Assessment Tool (CPAT), which requires officials to manually tabulate scores themselves (think of it like a math homework assignment). On the other end, there is COMPAS, which is only the tool we found that leverages artificial intelligence technology.

Except for COMPAS, the tools we discovered are transparent in the way they calculate their scores, meaning most scores can be calculated manually, with varying degrees of difficulty. But almost all RNA and pretrial risk trial tools we found use technology to automate their tabulations.

Latessa describes this as "machine scoring," which he says is similar in function to a calculator automatically adding up figures. "Machine scoring can simply mean you put the data in and it ticks out what you put in. … You know why the person scored the way they did because you collected the information and entered it into the system."

He contrasted this with the AI-based COMPAS where "you don't know what is being scored—it's taking data and it's calculating [a risk score] using some algorithm and giving you some prediction."

However, Chris Kamin, interim general manager at Equivant, notes the company is planning "to copyright the current COMPAS Core and Reentry assessments to make them transparent for the public to examine. Additionally, we are developing a very simple, transparent, point-additive version of the COMPAS that will be available in 2021."

Such efforts would bring COMPAS in line with other RNA and pretrial risk assessment tools, most of which do not use advanced, complex technology such as machine learning.

What can be complex, however, is how states implement these tools, how they determine whether they accurately work, and how they ensure they're being used as intended. While assessment tools themselves can have inherent problems, much of their impact comes down to the specifics of how and where they're deployed. We'll explore these issues further as we dive deeper into the world of risk assessment tools this week.