Well, they actually have already, but his latest behavior goes beyond the pale and constitutes a breach of basic judicial principles.
Not too long ago, United States District Judge Lynn Hughes made some rather sexist comments to a female Assistant United States Attorney. Ultimately, he was formally reprimanded by the Fifth Circuit.
He evidently has not gotten over it.
Without explanation, Hughes has banned this attorney from his courtroom, doing it in a very high handed fashion that left zero doubt that this is sheer retaliation.
This sort of thing cannot be permitted to go unchallenged.
The United States Attorney’s Office is pursuing this at the Fifth Circuit Judicial Standards and Conduct Committee. If Hughes does not back off, I think a formal suspension and notice to Judge that if this behavior continues, the next step will be a to formally request the House of Representatives to initiate impeachment proceedings on the grounds of abuse of power.
Alternatively, I think the other Judges of the court should pressure him to retire, as he is eligible for full retirement. His behavior reflects badly on the court as a whole and he just needs to leave now.