How do I referee a paper when the authors may be behaving dishonestly?
It is hard to tell from just your description alone (and without knowing the actual paper and revisions), but honestly this does not really sound like unethical behavior to me. Basically, what seemed to have happened was the following:


Authors hand the paper in. You review and claim that one of two important aspects is missing.
The authors disagree and, in an attempt to calm you over, write a short token paragraph about the second aspect (that they, in all honesty, don't believe to be very important).
You insist that the second aspect is super-important and needs to be covered more explicitly.
The authors give up and describe what the may think to be a minor aspect in much more detail than what they originally planned to do. Note that it is well possible that the authors were fully aware of this and had the data to do this all the time, but just did not think about it as something important. That they now agree that this is super-important sounds more like an attempt to win you over than anything else.


I have seen these kinds of exchanges play out in many, many journal submissions. Whether this actually improves the paper or not depends primarily on whether the second aspect was actually important or not. You say it is, but I have definitely seen reviewers get hung up on absolute sidenotes as well (which I am sure they honestly thought were essential), ultimately to the detriment of the manuscript.

More generally, it sounds a bit like you were a bit the "damned if you do, damned if you don't" kind of reviewer in this process. What kind of change did you expect after the second review round? You wanted this additional behavior covered, and they did - and now that they did, you wonder why they did not do so from the beginning and presume that something unethical is happening. If they wouldn't have addressed this point, you would have been unhappy because the authors did not fix your major comments.

What I generally try to teach my students is that when you review for journals, you need to recommend fixes that can actually be done. That is, there needs to be a reasonable and doable change that would actually make you happy to accept the manuscript. If you think that a paper can in no reasonable way be fixed, recommend rejection (and don't accept another review if the editor decides for revision). Never recommend revision if you won't like any outcome.

Edit based on Lohoris' comment to aismail's answer:


  I've no clue why is the other answer who got all the upvotes, this one seems reasonable and to the point, while the other one basically says "you suspect something, don't bother, go on, everything's nice" which is a wtf for me.


Accusing authors of fabricating the data is a serious suspicion and should not be raised lightly if there are other plausible explanations. It's not that you "shouldn't bother", it's that it simply does not sound like the data has been fabricated to me (or, at least, it does not sound significantly more likely than in any journal submission where the code etc. is not public, which is a systemic problem). There are any number of (good or at least less bad) reasons why the authors may now have data that they did not report on in the first version, including:


The authors generated this data in the many months between revisions
The authors had the data all along but did not consider it very important
The authors wanted to save up this data for another paper (either to salami-slice, or because they truly thought that those should logically be two separate papers)


It is widely accepted that even major changes to the data and analysis can happen in between major revisions, and indeed it is common-place at least in my community to add more data to the paper based on reviewer request. This is basically the reason why we do peer review in the first place - if we always remain suspicious when the authors do more than cosmetic changes, why do we recommend revisions in the first place?

I understand that the authors were talking about a "miscommunication" in the response letter, but frankly, after two revisions (especially with a very critical reviewer in the loop), most authors tend to write in the response letter whatever they think gives them the best chance to calm the reviewer over. I would not give too much weight to whatever people write in this non-public document.