How to detect when the "bibliography" of a paper has began?
Here are a few heuristics that could help solve the problem without ML:


If all the papers are from the Arxiv, you can download the source tex files and regenerate the paper without the bibliography
You can use Arxiv-Vanity or their code to render the document into HTML (or some intermediate representation), then use that hierarchical structure to easily remove the part with a bibliography
Look for the word "references" followed by high word density of the words "in", "pages" and years in the 1900s and early 2000s


I guess what I am saying is "the input already has a fairly rigid structure, why don't you use that instead of statistics?"