Certain Emails, not all are falling into SPAM
Receiving mail servers use a variety of different metrics to give your mail a 'spaminess' score to decide whether it is spam or not.


  Is there a process difference when mail is triggered by CiviRules or CiviMail rather then directly from the contact record?


There are small differences in the email headers generated via different routes. These may be affecting the score. The differing content is likely having an effect as well (even simple messages can appear spammy to receiving mail servers).

My hunch is that you don't have DKIM set up. Setting up DKIM will reduce your score and put you under the threshold in both cases, though it wouldn't work if:


your emails (headers and content) look really spammy to the receiving mail server's algorithms 
clients using the receiving mail server keep on marking your emails as spam


Typically a receiving mail server will add a header to the email with 'authentication results' that typically refer to DKIM, SPF and DMARC. For example, when google receives a mail, it adds a "ARC-Authentication-Results:" header.

Posting those headers would be useful so we can check what it says about DKIM and SPF:


If you are sending via mail() is it likely not using DKIM. The easiest way of adding DKIM is to switch to a third party mail provider.
Also, you may have (in the eyes of the receiving mail server) a mis-configured SPF record, so worth checking.


A couple of final things to note:


Each receiving mail server / service has its own algorithms. What works for one, might not for another.
You need to give the receving mail server time to get used to this new source of mail (regardless of whether the config is correct or not). If the total  volume of mail is low, receiving mail servers may never get used to it. This is one reason why many people choose to use a third party service.