Is it possible to set a timeout for optimization in PostgreSQL? In PostgreSQL is there a timeout when the optimizer stops the optimization process and returns the least expensive plan it has found so far? Is it possible to change its value?
No, and no, though it does have some heuristics (like join_collapse_limit) to limit how long it tries for, and it also has things like GEQO to switch to different optimisation strategies for big joins under some circumstances.

There's no optimisation timeout as such.

(It'd help if you'd explain the kinds of queries you're running and why they're taking so long to plan.)