Why does my house feel toasty when it's very cold outside, but cold when it's only a little chilly out?
It probably has to do with two things: Perception and mixing. 

When it's very cold the radiators are on more often. You feel the heat coming off them, which makes it feel warmer in the home even though the average room temperature is about the same. 

Also, when the radiators run less often in warmer weather, parts of the house (or each room) cool off faster than others. This results in cool pockets that you notice, if only subconsciously. Because the radiators aren't in use, causing convection and mixing, more temperature variation may occur.