Restrict block to specific pages is broken when using view pagination
It turns out since my block was itself a view, it received the page param from the url, and since pagination was turned on, it just went to other pages, which happened to be empty.

I fixed it by turning off pagination on the block view.