When the full U.S. Court of Appeals for the D.C. Circuit agreed earlier this year to consider the Consumer Financial Protection Bureau's constitutionality, it teed up the possibility of punting on that question and instead ruling on narrower grounds.

But the 11-judge panel that heard the case Wednesday appeared poised to take on the issue of whether the CFPB's independent, single-director structure runs afoul of the Constitution.

The judges, sitting in the court's ceremonial courtroom, focused squarely on the question of whether it was lawful, under the separation of powers clause, to restrict the president's power to remove the director “for cause” only. Cabinet secretaries can be dismissed for any reason.

Several judges appeared skeptical that they could strike down that structure, citing a 1935 case—Humphrey's Executor v. United States—in which the U.S. Supreme Court said a member of the Federal Trade Commission could not be fired at will by the president.

The arguments marked a pivotal point in mortgage provider PHH Corp.'s challenge to the CFPB. New Jersey-based PHH, represented by Gibson, Dunn & Crutcher partner Theodore Olson, prevailed before a divided three-judge panel that struck down the bureau's structure last year. But that split decision, authored by Judge Brett Kavanaugh, was later vacated by the D.C. Circuit's decision in February to grant an en banc review. The company is fighting a $109 million penalty—and many companies are riding on PHH's argument against the agency.

Olson sought to distinguish the CFPB from the FTC and other five-member commissions. He argued that the CFPB's power is “all vested in one person as opposed to being distributed among several people,” and later, he described the agency's director as “completely independent of the president and completely unaccountable to the president.”

What follows are highlights from the 90-minute hearing.

The D.C. Circuit will likely hit the constitutional question. No punting there.

In the October panel decision, the three judges were united in overturning the CFPB's interpretation of the Real Estate Settlement Procedures Act, the statute at the foundation of the agency's enforcement action against PHH. But Judge Karen LeCraft Henderson, in a dissent, said the court should not have reached the constitutional question about the CFPB's structure.

The full D.C. Circuit asked in February for arguments on three questions, among them: “May the court appropriately avoid deciding that constitutional question given the panel's ruling on the statutory issues in this case?”

There was no talk of taking that off-ramp on Wednesday, a suggestion the court is prepared to confront the constitutionality of the single-director structure head-on.

Some judges were skeptical the CFPB's structure intruded at all on presidential power.

Judge Patricia Millett appeared unconvinced that the CFPB's structure encroached executive power any more than an independent commission. She noted that no more than three commissioners at five-member agencies—such as the FTC and the U.S. Securities and Exchange Commission—can be from the same party, forcing the president to occasionally appoint members from the opposite political party. With the CFPB, the president does not face any such restriction, she said.

Posing the question of which was the “greater intrusion,“ Millett asked whether it was more limiting for a president to choose “someone of your choice for the bureau or someone of the other party for the commission?”