CANUSB SJA1000 Filtering
I suspect that you are the victim of a deeply, deeply flawed manual. I notice the "r" and "R" commands, which one would think are read commands, are listed as read commands but described as transmit. Furthermore, it states that 11-bit ids can be selected over the entire 11 bits, but not 29-bit ids. This is an obvious garbling of the function of the SJA1000 in basic mode, where only the 8 msbs of the 11-bit id field can be filtered. Even allowing for English as a second language, this is not a good sign.

You have not stated it, but I assume you're testing involved hooking up to an existing CANBus bus and reading what's on it. Furthermore, I suspect that the existing units all use the standard (non-extended) ID field with 11 bits, and you are using the "riii" command to read it. 

The first thing you need to realize is that the manual makes no sense. Compare Fig.9 and Fig.10 in the NJA100 manuals. For 11-bid ids the id data is contained in ID 28-18,with mask registers 0 and 1 being used. For 29-bit ids, the 11 lsbs are ID10-0, and the mask registers corresponding are 3 and 4. It appears that the manual is using the R command, and then somehow pretending the id is formatted as if it were read using the r command, if that makes sense. The claim that using E0 instead of F0 to mask out the RTR bit is confirmation of this.

More confirmation comes from your id data. Your mask setup did not discriminate in the high bits, and I notice that all 10 id bits you show have acivity - that is, they are responding to don't-cares in the mask.

The one thing you've done right (I think) is left-shifting your mask bits, but not because of RTR considerations, but in order to left-justify the 11-bit id in the 12 left-most bits of the registers. 

So I'd try M3ACFFFFF and m000FFFFF instead. If that doesn't work, try setting various combinations of bits in the AC0 and AC1 registers, and see ids you detect.