PostgreSQL skips WITH query
That is because the main query doesn't reference division_by_zero anywhere.
The documentation says:

[...] PostgreSQL's implementation evaluates only as many rows of a WITH query as are actually fetched by the parent query.

That is zero in your case.
If you want the CTE to be executed, you could for example add a WHERE condition like
WHERE EXISTS (SELECT 1 FROM division_by_zero)