How to repair (sand down) the sides of a paperback book properly?
When working on the cut edges of a book you need to clamp the pages of the book closed between boards. Making a 'board book board' sandwich.
Ideally the boards are as big as the pages of the book and you can clamp them just around the book with separate clamps.
Which in turn you can clamp down in a vice, to hold it for you working on it.
It is also possible to skip the clamps and position the book in its boards in a vice but it is harder to position them right in that way. Even when for working you should not have clamps, it is often good to clamp the work before placing it in the vice and take the clamp off when it is safely in the vice.

If your boards are bigger or smaller than the front and back of your book, you can still use them but you have to be careful to position them such that the edge of the book you want to work on is. supported and not work on any part which is not supported.

If you want to take a bit off the book, you can have the book stick out of the boards a little, but often it is easier to take a bit of the board along with the bit you want to take off the pages.

When the pages are clamped in like this, they should not be able to puff up, but you should still work in such a way the paper of the book can not get wet at all.

If with this way of working you still have problems with 'puffing' please add photos and a good description of the problem.

The book binders I have seen working all used a knife to cut the pages to size, with the books clamped in between boards, not using sand paper, but they do take off more than you seem to want.