What constitutes plagiarism when writing a textbook?
It's the words.
Speaking as a mathematician the formulas and calculations are all common knowledge, even if it's new to undergraduates. But the words you use to explain it need to be your own.  The style you use will uniquely identify you.  For examples check out Carl sagan's cosmos and Stephen Hawkins a brief history of time. Beautifully worded explanations of complex mathematics.