Can I swap out this 20A breaker for a 15A breaker?
That's fine.  The breaker protects the wire, and it's always allowed to use larger wire than you need, so bumping the breaker down to 15A is legal.  Based on your anticipated load, it sounds like you won't have a problem with the breaker tripping.  

The one issue you may have is if you intend to run a space heater on one of the shed outlets sometimes -- they usually use a whole 15A circuit by themselves, so that wouldn't be compatible with also having the lights on the circuit.  However, that would just trip the breaker and wouldn't present a safety issue.  

Note that since you have pool lights on this circuit, it'll need to be GFCI protected.  If your current 20A breaker is a GFCI breaker, replace it with a 15A GFCI breaker.  If it's a normal breaker currently, you probably have a GFCI somewhere downstream (though check this, and rectify if not), so replacing with another regular breaker is probably okay, but doubling up on the GFCI is allowed, so you're welcome to use a 15A GFCI breaker in either case.