I'm in the process of making changes to my site so that we can be a SAML 2.0 Service Provider. We will be doing IdP initiated SAML with Out-of-Band account federation.

My question is this: Given a SAML response that is posted to the target page on my site, how do I verify that the response was generated by a trusted source?

I can validate the signature that is embedded in the response, but... couldn't anyone just sign an response and start posting it at my site, attempting to validate users? I assume there is something in the signature that I can extract and match against a list of trusted Identity Providers? But couldn't an attacker just spoof that?