We Build the Wall founder Brian Kolfage was never technically removed from his post as director, meaning he could stand in for the company at its upcoming criminal trial, a Manhattan judge said Thursday.