How to interpret this rejection email from Journal of American Math Society? Anything to read between the lines?
There isn't anything between the lines.  It's simply a rejection. The editor and reviewers feel that your paper is clearly not strong enough for this journal, and they're telling you so.

Note that JAMS is an extremely selective journal, generally considered one of the top four out of all the many hundreds of journals in mathematics.  It is very very hard to get a paper published there.  Most of the papers they publish will be major breakthroughs in an area, or solve a famous open problem.  The "10%" is actually misleadingly high, because the vast majority of papers are never submitted there in the first place because the authors know they have no chance.

Such journals often do a quick first round of reviews, where the reviewers are only asked if they think the results are potentially important enough to warrant publication in JAMS.  Only if they say yes does the paper go on to a full technical review.  In this case, it sounds like they said no.  You didn't receive a detailed report because they didn't need to do a detailed review of the paper to make a decision.  Remember that their job is ultimately to make the decision; it's not their job to give you feedback or help you improve your paper.  Sometimes that happens along the way, but when the decision is clear, they won't spend time on a paper that, in their view, can't be made publishable in this journal.

The reviewers can, as in this case, make comments about things they happened to notice as they were doing their quick review.  That's why you got the note about the reference.    You'll have to read the reference to determine exactly what was meant here; it may be that the reviewer thinks that paper may already contain your results, or at least that they are similar enough that you need to carefully explain the difference.  But it sounds like the reason for rejection was the overall significance of your paper, and this was just something they noted in passing.

How to move forward: submit to a different journal.  If you want feedback on your paper first, you'll have to get it from someone else, e.g. a mentor, PhD advisor, collaborator or fellow researcher.  Such a person could also help you get some ideas of journals for which your paper might be better suited - where it would have a better chance of acceptance or at least a full review.