What does does $O$ mean in this context?
It means exactly the same thing.  It means there is a constant $c$ such that the term is $\le c \cdot \epsilon^2$ (where $c$ does not depend on $\epsilon$).  Big-O notation is not unique to computer science; it is also widely used in mathematics.  See https://en.wikipedia.org/wiki/Big_O_notation, which explains some ways it is used in mathematics.