Can I use a stamped arxiv paper as a proof of priority if it has been rejected?
I recommend against it. You can put online anything you like and use it to claim priority, but if the paper is hosted on your personal website then you lose the credibility that the date stamp of an official arXiv-hosted paper comes with (since just the stamp itself would be easy to forge), so in the event of a priority dispute this would not be a foolproof form of evidence.

Also, as has been discussed in other questions on academia.se (can't look it up right now), people who are new to academic publishing worry excessively about their ideas being stolen by others. Especially given that your manuscript was rejected by arXiv, the chances of this happening are effectively zero. You would be much better off using your energy to write a good manuscript rather than worrying about datestamps.