Controversial New Haven attorney Norm Pattis might have to return the bulk of a $250,000 retainer, if the plaintiffs in a new lawsuit get their way.