How do I know if it's worth restoring my bike?
I volunteer at a community bike shop. We take old donated bikes and fix them up for sale, so I have a lot of experience with this exact dilemma. Here are a few reasons why I will stop working on a bike I am refurbishing:


Frame has worse than just surface rust: i.e. extensive pitting and / or holes.
Seized parts, especially if they need replacing. Sometimes a seized seatpost isn't so bad, as long as the bike is sold along with that information. However if the bottom bracket needs service and can't be removed, the frame is scrap. The same goes for quill stems that can't be removed, locking the fork into the frame.
Damaged threads. If it's not possible to rethread an interface that has been wrecked (e.g. cross threaded), scrap the frame.
Bent or cracked tubing. The odd dent is fine, but if a bike has obviously been bent in an accident I scrap the frame. Same goes for tubes with any cracks in them, even if it's fixable. Obviously any carbon frame with cracks is worthless.


I'd also add that the original quality of the bike plays into these determinations as well. I will spend a lot less time working on the seized seatpost of an old BSO.

I'll also point out that a lot of these issues can be solved with extreme effort, e.g. replacing tubes in the frame. It just isn't worth it in this context and I suspect the same guideline applies for most folks tinkering at home.