How common is it for journal to reject a solicited paper?
This is not common, but it certainly does happen.  I know a small number of people who have had their invited submissions rejected.  Whether it is a likely possibility in particular instance will depend on how many invitations the editor(s) send out—and, correspondingly, how well they know the potential contributors and their work.  If the invitations are kept to a relatively small group of individuals familiar to the editor, it is less likely that anyone will submit a manuscript that is of low quality or is not topical.

Whether this is likely to happen also depends on how the submissions are being evaluated.  Sometimes, an expert guest editor for a special issue may take care of all the refereeing themself.  However, this will not work if the topic is too broad or there are too many submission.  I was actually responsible for getting a low-quality invited paper rejected from a special issue once, when I was asked to referee it.  The work was simply too shoddy to merit publication in any venue.