A federal judge in Los Angeles set the government's criminal case against Tom Girardi for May 21.