Optimizing a heating system
It sounds like what you want is a set of heat loss calculations. These are the calculations a heating contractor would do in order to determine how much power (BTU output) a furnace/boiler would need in order to keep your house warm. A heating system is then sized such that it can overcome the heat loss your home would experience on the coldest day of the year.

These calculations are based on the area of each room, exterior walls, number of windows and doors, type of insulation, etc. I saw an episode of Ask This Old House that included a heat-loss analysis using software that calculated this using a house model / floor plan entered by the contractor. According to this show page (http://www.thisoldhouse.com/toh/tv/ask-toh/products-and-services/episode/0,,20321333_20311273,00.html), that software was made by WrightSoft.

Unfortunately, this software seems to be aimed at contractors rather than homeowners, and has a price tag of $200-$500. It is also probably rather technical. There are some other options, including some free online heat loss calculators. But why bother, when you can probably have a heating or insulation contractor come to your house and do these calculations for you?

In Massachusetts, where I live, the utility companies offer an energy audit service where, among other things, they inspect and measure your house and offer you various options for adding insulation. For each option, e.g. blowing cellulose insulation into the exterior walls, they also list the estimated savings per year, based on local energy prices. Using that, you can calculate how many years it'll be before the insulation work pays for itself. In my case, about $2000 of insulation work would pay off in between 2 and 3 years.

If utility companies in your state don't offer this service, you may still be able to find an insulation contractor who can. And for figuring out which doors to close to reduce heat to unused rooms, why not do some experiments? You can buy a thermometer with multiple remote sensors, and watch how the temperature fluctuates in your unused rooms.