Papers on IEEE Xplore or ACM DL
Does this count as an IEEE Conference paper?

Yes

Can I safely assume that all papers available on IEEE Xplore or ACM
Digital Library are published as IEEE transaction or conference
papers, and ACM papers respectively?

I think so and will be surprised if that is not the case.