Erning.write() » spambayes

Filter spam and virus

erning — Mon, 27 Oct 2003 21:46:49 +0000

I’m currently working on a large email system which will serives millions of mailboxes – like hotmail.com.

Except sending/receiving emails, the most important feature is spam and virus filtering. I’m using SpamBayes for spam filtering and AnomySanitizer plus ClamAV for virus filtering. They look working well.

Here’s a Chinese document about how to configurate SpamBayes, AnomySanitizer and ClamAV on an exists email system.

The next problem is to make the email system effective and scalable. Anyone has suggestion?

Server Site Spambayes II

erning — Thu, 25 Sep 2003 21:38:55 +0000

Since everyone has a different idea of what is spam. The spam filter with a global database will filter some useful emails. It’s not acceptable for customers although there’re only 1% useful emails filtered as spam. So we individual spam databases for each mailbox and let the customers to train the spam filter themselves.

Three buttons was added to the web mail. ‘Delete as Spam’, ‘Recover from Spam’ and ‘Reset Spam Filter’. It’s similar to SpamBayes plug-in for Outlook.

I’ve configured the solution on a production server. Below is the summary of the e-mail server’s log summary for Sep. 25,


Grand Totals
------------
messages

  87812   received
  56351   delivered
   2012   forwarded
    606   deferred  (5096  deferrals)
   1342   bounced
  54150   rejected (49%)
      0   reject warnings
      0   held
      0   discarded (0%)

   3486m  bytes received
   3816m  bytes delivered
  13591   senders
   4993   sending hosts/domains
   6937   recipients
   1750   recipient hosts/domains

Server Site Spambayes

erning — Sat, 20 Sep 2003 21:18:26 +0000

Installed and tested Spambayes on one of the productive email server of our company. It works very well. About 30,000 (around 1GByte) junk emails are filtered per day.
The email server hosts more than 2,500 domains and 30,000 mailboxes.

Currently, we use global spam recognition database and only drop the very certain junk emails. I think it’s better to setup database for each mailbox and provide an user friendly control panel.

Anti-Spam

erning — Sat, 26 Jul 2003 21:15:12 +0000

Do you hate spam? I do. I get hundreds of spam a day! I’ve tried several methods to stop spam. Outlook rules, procmail regex filters, black/white lists and others. But all of them are not clever enough. After tried SpamBayes, I believe the filters can kill spam.

I’ve installed a spam filter plug-in (SpamBayes) for my Outlook 2002. I just gave 10 spam emails and 10 non-spam emails to the filter. After the training the filter works very well. And I was able to disable the stupid Outlook’s rules.

But it still have a problem. I have to receive all mails and filter them locally. It should be better to recognize the spams at the server-side. So I decided to setup a server-side filter. Fortunately, SpamBayes works well with procmail.

Our email system is powered by the last stable postfix. I am very care about secure issue, so SMTPs, POP3s, IMAPs and HTTPs have been configuated for sending, receiving and reading email. To enable the SpamBayes filter, I just add a few rules to the procmailrc file. Now, all coming emails will be handled by SpamBayes first and delivered to different folders. Spam will be delivered to “Spam Certain” folder, unsure will be delivered to “Spam Unsure” folder and ham will be leave in the Inbox.

SpamBayes’ accuracy is dependent upon having a “sufficient” sample from which to make its decisions. I have to teach SpamBayes which is spam and which is not. There’re two special folders named ‘Filter Train/Ham’ and ‘Filter Train/Spam’ in my mailbox. I wrote a small script to let SpamBayes train from the two folders and empty them after a successful training. Copy the wrong delivered emails and unsure emails to the Ham and/or Spam folders. SpamBayes will train and handle them correctly next time.

Everyone has a different idea of what is spam. So we have individual spam databases for each user.

Now, guys use our email system are able to receive non-spam emails to local mailbox using POP3s and train the filter using IMAPs or the webmail. It saves the time, bandwidth and money.

Next, I am trying to find a better anti-spam solution for large scale. For instance, OnlineNIC Inc. the company I am working for currently. It hosts thousands of virtual hosts, email boxes. The solution should be clever, effective, customizable. And it should be able to filter both incoming and outgoing emails.

Resources,
SpamBayes
Will Filters Kill Spam?
OnlineNIC Inc.