Why does my house heat up, even when it's cool outside?
Meet solar gain

For the most vivid example of solar gain, sit in your car with everything off - A/C off, blower off, windows rolled up tight, doors shut tight.  You can't. Your body will force you to open a door or something because it will become unbearably hot within minutes.  

If it were always night, or if your house were entirely in shade, this would not be an issue.  However, your house is in sun,  and that means it is being actively heated by about 100 watts (300 BTU/hr) per square foot (around 1 KW/m²). 

That's really a square foot directly facing the sun, so it'll be less if you're not at the equator at noon... but walls count too, so that's kind of a wash. The reflectivity of the roof and walls help.  So let's say all in all, 50% gets through. 

This is why a car with 48 square feet (4.5 m²) of cabin gets so hot so fast - it's intaking 2400 watts/8000 BTU/hr of heat.   That's the heating power of an oven.  This is why leaving a baby in a car is so serious. I'm not equating a house to a car, just pointing out that solar gain is not to be armwaved or ignored, and is definitely your #1 load by a wide margin, maybe even by an order of magnitude. 

So, your 1000sf (93 m²) house, is absorbing 50,000 watts of heat, or about 150,000 BTU/hr, from solar gain.  

Yes, this is a stupidly large amount of energy.  Too bad we can't harness it for something, eh? :) 

Insulation slows it down. But only slows it.  Given enough time, it still gets through.  

Thermal "mass" works both for and against you.  The thermal mass inside the insulation envelope helps your house resist changes in temperature.  However the thermal mass outside the insulation envelope (roof, joists etc.) has been warmed by the sun all day, has reached 120-140F (50–60°C), and holds that heat for a few hours after sunset.  That means its high heat is still trying to push through the insulation even after the sun has gone down.  Conversely in the morning, even with full sun, you get some relief before the roof and joists heat up.  

At least it's more efficient...

Your A/C unit interchanges with air, and in your case, you are saying the ultimate heat sink is cooler than the thing it's cooling.  That gives a heat pump an advantage; it is pushing heat "downhill".  You can check it with a load monitor (if the cycle length doesn't make it obvious), but you probably aren't using near as much power as on a hot-air day.

It's even better, of course, to interchange heat with groundwater. But that is a more specialty unit.