“We walked into a hornet’s nest I didn’t even know existed,” Mr. Stephens said.
In response to the protests, the state commission recommended a much simpler setup based on software already used by the state courts. But even this algorithm is difficult for a layperson to understand. Asked to explain it, Mr. Stephens suggested speaking with another commissioner.
Nyssa Taylor, criminal justice policy counsel with the Philadelphia A.C.L.U., was among the protesters. She worries that algorithms will exacerbate rather than reduce racial bias. Even if governments share how the systems arrive at their decisions — which happens in Philadelphia in some cases — the math is sometimes too complex for most people to wrap their heads around.
Various algorithms embraced by the Philadelphia criminal justice system were designed by Richard Berk, a professor of criminology and statistics at Penn. These algorithms do not use ZIP codes or other location data that could be a proxy for race, he said. And though he acknowledged that a layperson couldn’t easily understand the algorithm’s decisions, he said human judgment has the same problem.
“All machine-learning algorithms are black boxes, but the human brain is also a black box,” Dr. Berk said. “If a judge decides they are going to put you away for 20 years, that is a black box.”
Mark Houldin, a former public defender who was also among the protesters, said he was
concerned that the algorithms were unfairly attaching labels to individuals as they moved through the criminal justice system.
In an affidavit included with a lawsuit recently filed by the defender’s office, a former Philadelphia probation officer said the probation department’s predictive algorithm also affected arraignment hearings. For years, she said, if someone was arrested and charged with a new crime while on probation — and had been deemed “high risk” by the algorithm — the probation office automatically instructed the jail not to release the person.
A spokesman for Philadelphia County denied that the system operated this way. “Every detainer issued is reviewed by supervisory staff of” the probation department, he said, “and notice is sent to the appropriate judicial authority.”