Dirrell was a standout as an amateur and won the 2003 United States national amateur championships at middleweight.
He won the middleweight bronze medal for the United States at the 2004 Olympics in Athens, Greece.