Is it possible to set a timeout for optimization in PostgreSQL?
No, and no, though it does have some heuristics (like join_collapse_limit) to limit how long it tries for, and it also has things like GEQO to switch to different optimisation strategies for big joins under some circumstances.

There's no optimisation timeout as such.

(It'd help if you'd explain the kinds of queries you're running and why they're taking so long to plan.)