Little is known about his early life, but he was probably born west of Chartres, perhaps in the town of Brunelles, near to Nogent-le-Rotrou, making him one of the first of the Netherlandish composers who was actually French.
A Jachet Brumel was organist for the Ferrara court in 1543, and is presumed to be Antoine's son.