Statues and busts of the 19th century U.S. Chief Justice Roger Taney have been taken down recently in his native Maryland, but at the U.S. Supreme Court, depictions of the author of the notorious Dred Scott decision are still visible—and not likely to disappear anytime soon.

Before and after the Charlottesville, Virginia, violence this month, Taney has been lumped together with Confederate generals and leaders as symbols of racism and slavery. Taney statues were removed this month from public places in Annapolis and Baltimore, while a bust was removed from Frederick, Maryland.

In the 1857 ruling, Taney wrote in Scott v. Sandford for a 7-2 majority that African-Americans were not citizens, and had been viewed as “so far inferior, that they had no rights which the white man was bound to respect; and that the negro might justly and lawfully be reduced to slavery for his benefit.”

A marble bust of Taney, along with busts of all chief justices through history, is displayed in the Great Hall located in front of the court's chamber. Taney's portrait hangs in the court's oak-paneled East conference room, also accompanied by paintings of other chief justices.

Listen now to our free webcast: Law Firms Respond to Charlottesville

“The portrait should stay for historical purposes, but the bust should go,” said civil rights lawyer Barbara Arnwine, founder of the Transformative Justice Coalition.

She said the Scott ruling was “not a reluctant decision” on Taney's part, and his language helped perpetuate the “severe subordination” of African-Americans. To this day, Arnwine said, “White supremacists cite Dred Scott all the time. It's their favorite. He doesn't deserve veneration.” She also noted that because of the decision, members of Congress balked in 1865 at putting a bust of Taney in the court's chamber, then inside the U.S. Capitol building.

Here's how Taney's successors have commented on his legacy: Chief Justice Charles Evans Hughes called the Dred Scott decision the court's greatest “self-inflicted wound.” And the late Chief Justice Warren Burger once described the day when Taney announced the opinion as “a dark day that cast a cloud for a century over the reputation of a great justice.” The court's public information office had no comment.

In an informal survey, other lawyers weighed in:

“Because the only criterion for having a bust and portrait is serving as chief justice, I favor keeping them,” said Steve Klepper, a Maryland appellate practitioner and blogger, adding that he thought that taking down Taney statues in Maryland was appropriate. “We have to wrestle with the fact that one of the most prominent jurists of the 19th century was the author of an intellectually dishonest racist screed.”

Columbia Law School professor Jamal Greene said removing the bust or portrait from the court would “oversimplify the [Dred Scott] decision and worse, tends to call attention away from the ways in which the decision reflected ugly truths about our constitutional history.”

Josh Blackman, a conservative Supreme Court blogger who has written about the Taney dilemma, said, “The Great Hall has busts of all the chief justices, not because of any particular opinions they wrote, but because of the position to which they were confirmed. Taney belongs in that crowd.”

More broadly, he added, “We should hesitate before assessing all of the justices based on how their opinions have stood the test of time.” Blackman cited among other cases the 1927 decision in Buck v. Bell, which favored eugenics and was written by the otherwise respected Justice Oliver Wendell Holmes.

Another indicator of how the court might react to controversy over Taney's bust in the court building is how the justices handled another unpopular sculptural feature in the court building. In 1997, several Muslim organizations requested that one of the friezes that line the upper walls of the court chamber be altered. A Muslim lawyer had noticed that the prophet Muhammad was shown along with other lawgivers including Justinian and Napoleon. Depictions of Muhammad are generally prohibited by Muslims, and the groups wanted his image sandblasted off the frieze.

Then-Chief Justice William Rehnquist turned down the request, but the court made one small concession, changing the wording of its fact sheet on the friezes to state that the depiction was meant “to honor Muhammad and it bears no resemblance to Muhammad. Muslims generally have a strong aversion to sculptured or pictured representations of their prophet.”