The family originated from the county of Ponthieu.
The last generations of the House of Brimeu, intermarried with important Flemish noble families like the houses of Glymes, Croy, Ursel, Schetz, Van de Werve, Snoy and Tucher von Simmelsdorf.