   Content-based Spam filtering - losing battle?
  It seems to me that spam filters(not to be confused with blacklists and whitelists) could not keep up with latest spamming methods and are going to be defeated in foreseable future, or the rate of false positives (non-delivered emails) is going to increase. These days, some spam emails include relatively long text quoted from literature, encyclopedia, or just random site. It is obvious that naive techniques such simple sum of "spammness" indexes assigned to different words (so-called "Bayesian filtering") can not keep up even with most basic techniques such as addition of many random words. Automatic "training" only makes things worse. I do not see how anything could be done with spam that (for example) contains large randomly selected piece from classic literature (or wikipedia, or whatever else), with few links in the middle.In a way, false positives is much worse for many of us [who have private email that they don't disclose everywhere] than false negatives. The worst thing is that there is often no "delivery failed" notification whatsoever...So, are we doomed? discuss.btw: spam also gets weirder and weirder... I had blog on my website, and i'm now seeing fake referers in the server logs even that blog is long gone. Blog didn't show referers, by the way[Edited by - Dmytry on December 6, 2005 3:54:16 AM]
Quote:Original post by DmytryIt seems to me that spam filters(not to be confused with blacklists and whitelists) could not keep up with latest spamming methods and are going to be defeated in foreseable future, or the rate of false positives (non-delivered emails) is going to increase....So, are we doomed? discuss.I doubt it. A while ago I registered a spam e-mail address @gmail.com and deliberately posted it around a few forums. I also signed up for a few mailings as a test case. Within about 72 hours I had my first unsolicited e-mail, but it went straight to the Spam folder. Google correctly identified every message that was spam.It did have one false positive, an e-mailed weekly newsletter from Men's Health. I suspect it tripped up over the aberrant concentration of sex-related articles ("Boost Your Libido!", "How To Rev Her Up In Bed", "Herbal Supplements -- Quackery or Quality?") that were in that particular newsletter, since it didn't block the others from the same series. (I also took the step of clicking on the letter and marking it as "not spam", which probably strongly weighted the decision about whether the next letter was spam.)Quote:Automatic "training" only makes things worse. I do not see how anything could be done with spam that (for example) contains large randomly selected piece from classic literature (or wikipedia, or whatever else), with few links in the middle.Context, of course. The probability that a link to Viagra will occur in the middle of an actual Shakespearean soliloquoy is virtually nil. "To be or not to V1AGR4, that is the question..."   ;  I've been using POPFile in Outlook and it's given me great results. It's a bayesian filter so you have to train it but it doesn't take as much work as you would think. Since Oct 8th I've gotten 56,304 email messages, and POPFile filtered 94.15% accurately. Works well enough for me...   ;  I've never recieved one single spam mail on my hotmail account (which I use for various stuff where I dont want to give out my normal mail address). I have, however, had problems with it dumping legit mails into the spam folder (and the spam folder is emptied every day). So now I've set the spam filter to "low". Still no spam though, the only difference is that hotmail keeps bugging me about "your spam filter is set to low blah blah".   ;  I'd like some sort some sort of whitelist/blacklist based on digital signatures...  So that signatures which you trust will never be counted as spam. The problem here is that not many people actually use digital signatures [sad] ;  I think the future of spam filtering is through networking. People are the most effective spam filters in existence at the moment. If you can detect when a group of messages share similar characteristics (e.g. all start with broken text and then continue with correct english, all have a particular link or image attached), and you can recieve reports from some users that the messages are spam, then you can junk the messages for the rest of the users.I have a sneaking suspicion that this is what GMail does.   ;Quote:Original post by DmytryIt is obvious that naive techniques such simple sum of "spammness" indexes assigned to different words (so-called "Bayesian filtering") can not keep up even with most basic techniques such as addition of many random words. Actually, Bayesian estimation is not the problem, it's the way it's used. Bayesian estimation is the basis for the whole of formal estimation theory, control theory, large parts of robotics, and other fields of engineering. It is a very powerful tool because it lets you quite literally get information from "nothing".You are correct, though, that it is difficult when just applied to get the "spammness" of each word, since each word by itself does not constitute a "spam" message. Unfortunately, there aren't many better ways to apply it to spam filtering.   ;  I just use a simple ruleset run one at a time in order.1. Move all mail (that just came in) to junk folder.2. Move all mail adressed to me and others to MassMail Folder.3. Move all mail adressed just to me to Personal Folder.4. Move all mail for (adress list) to PersonalI end up with very few letters I want in junk, maybe one a month, and very little junk, maybe four or five a month.   ;  I use Thunderbird's built-in baysian filter. I get quite a few mails. On average I end up with one spam message in my inbox. The last time it marked a legit message as spam was in the week I installed it (when the filter wan't trained yet).What really surprizes me is that there's still a large ammount of spam sent out in the first place. Paying a hacker with a botnet to send out a spamrun of a million or so can't be cheap so the ROI must be pretty good on spam runs.   ;Quote:Original post by kSquaredQuote:Original post by DmytryIt seems to me that spam filters(not to be confused with blacklists and whitelists) could not keep up with latest spamming methods and are going to be defeated in foreseable future, or the rate of false positives (non-delivered emails) is going to increase....So, are we doomed? discuss.I doubt it. A while ago I registered a spam e-mail address @gmail.com and deliberately posted it around a few forums. I also signed up for a few mailings as a test case. Within about 72 hours I had my first unsolicited e-mail, but it went straight to the Spam folder. Google correctly identified every message that was spam.It did have one false positive, an e-mailed weekly newsletter from Men's Health. I suspect it tripped up over the aberrant concentration of sex-related articles ("Boost Your Libido!", "How To Rev Her Up In Bed", "Herbal Supplements -- Quackery or Quality?") that were in that particular newsletter, since it didn't block the others from the same series. (I also took the step of clicking on the letter and marking it as "not spam", which probably strongly weighted the decision about whether the next letter was spam.)Quote:Automatic "training" only makes things worse. I do not see how anything could be done with spam that (for example) contains large randomly selected piece from classic literature (or wikipedia, or whatever else), with few links in the middle.Context, of course. The probability that a link to Viagra will occur in the middle of an actual Shakespearean soliloquoy is virtually nil. "To be or not to V1AGR4, that is the question..."But how you can automatically determine context? Have database of all literature, encyclopedias, webpages, etc? Isn't it going to have insane false positive rate?I'm talking of spam like this:To be or not to be, that is the question.  Whether 'tis nobler in the mind tosuffer The slings and arrows of outrageous fortune Or to take arms against a....Of course real spammer aren't going to use well known texts, he could use random pages from the web.Also: there is still a possibility for large email providers to use non-filtering approach, such as blocking links that are sent to too many users (sort of blacklisting), or not letting same IP to mail too many users. You can't say for sure that google blocked emails using content filter... Google has a lot of other information avaliable besides content.Another possibility is to crawl the page linked from email and use content from there, this indeed could work even if email looks okay.[Edited by - Dmytry on December 6, 2005 3:11:26 AM]