How to report to journal when work published in it has already been done by some other researcher?
For the case of the "forgotten" reference to previous similar work, I'd say you should assume good faith: maybe they just missed it. If it's not plagiarized then it's not really fraudulent. Re-discoveries happen all the time and it's not generally expected that all papers cite all the relevant literature. It can however be frustrating to the authors of the original paper.

If your intention is to notify the community that a very similar work exists, which can be very useful, then you could write a letter to the editor. These are often published if the journal find they are of value. Word it without hinting that you suspect the omission was made on purpose as this will not achieve anything.

For the large number of citations to seemingly unrelated papers from a given author, again it might just be a coincidence. But you cannot exclude a "citation ring" scheme. What you can do is communicate your concerns privately to the journal. If you gather enough evidence of an organized fraud you can also make it public through post publication review-websites (pubpeer.com, etc.). Do not hold your breath though, it's very possible that it will have no consequences whatsoever.