[gentoo-user] RFC: Implementing a spamfiltering frontend

[gentoo-user] RFC: Implementing a spamfiltering frontend

[Pandu Poluan]

Hello list!

Due to the increase of spam/phishing emails received by my office, I
decided to explore the idea of implementing a spamfiltering 'frontend'
in front of my email server.

Here's how I plan to do it:

fetchmail (G) --> postfix (G) --> amavisd+spamassassin+database (G)
--> postfix (G) --> current email back-end (WS) --> clients (W)

(G) = the single Gentoo server working as mailfilter
(WS) = mail server on Windows Server
(W) = various Windows clients (XP and 7)

I need fetchmail because currently we still use a hosting company, at
least until August when we host everything on our own. Then, we'll
drop fetchmail and expose postfix for the world to deliver the mails
to.

I also want to use amavisd's 'pen pal feature', so the outgoing emails
will take this route:

clients --> mail back-end --> postfix --> amavisd --> postfix --> ISP
mail-relay server

Now, did I plan right? Can anyone poke any holes in my plan before I
tie a 'ganbatte!' headband and study hard to implement the system
successfully or die trying?

Rgds,


-- 
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Andrea Conti]

On 21/05/11 06.13, Pandu Poluan wrote:
> Hello list!
> 
> Due to the increase of spam/phishing emails received by my office, I
> decided to explore the idea of implementing a spamfiltering 'frontend'
> in front of my email server.
> 
> Here's how I plan to do it:
> 
> fetchmail (G) --> postfix (G) --> amavisd+spamassassin+database (G)
> --> postfix (G) --> current email back-end (WS) --> clients (W)

Having a second postfix instance between amavisd and the email server is
going to make things way more complicated. Amavisd is perfectly capable
of speaking smtp/lmtp by itself, so unless you need to perform complex
mail routing you could directly send the filtered mail to the windows
server.

Other than that, I have very similar setup
(fetchmail->postfix->amavis->cyrus-imap, where all hops but the first
are done with lmtp) that has been working quite well for the past few years.

HTH,
andrea

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Pandu Poluan]

On 2011-05-21, Andrea Conti <alyf@alyf.net> wrote:
> On 21/05/11 06.13, Pandu Poluan wrote:
>> Hello list!
>>
>> Due to the increase of spam/phishing emails received by my office, I
>> decided to explore the idea of implementing a spamfiltering 'frontend'
>> in front of my email server.
>>
>> Here's how I plan to do it:
>>
>> fetchmail (G) --> postfix (G) --> amavisd+spamassassin+database (G)
>> --> postfix (G) --> current email back-end (WS) --> clients (W)
>
> Having a second postfix instance between amavisd and the email server is
> going to make things way more complicated. Amavisd is perfectly capable
> of speaking smtp/lmtp by itself, so unless you need to perform complex
> mail routing you could directly send the filtered mail to the windows
> server.
>
> Other than that, I have very similar setup
> (fetchmail->postfix->amavis->cyrus-imap, where all hops but the first
> are done with lmtp) that has been working quite well for the past few years.
>
> HTH,
> andrea

Hmm... interesting points. But can it still do the 2nd part of the
equation, that is, perform outgoing routing?

I had thought that I'll need amavisd to re-inject the email to postfix
so that postfix can do mail routing, i.e., mails to my domain get
routed to the Windows server, while outgoing emails get routed to my
ISP's mail relay.

Of course I can have postfix to skip amavisd for outgoing emails, but
then I guess I'll lose amavisd's automated whitelisting (the so-called
'pen pal' feature).

Rgds,
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Andrea Conti]

>> so unless you need to perform complex
>> mail routing you could directly send the filtered mail to the windows
>> server.

> Hmm... interesting points. But can it still do the 2nd part of the
> equation, that is, perform outgoing routing?

That's what I meant with "complex mail routing" :)

The problem with having two passes through postfix in the mail routing
chain is that you either run two separate postfix instances with
independent configurations or you have to figure out a robust way to
avoid loops.

It can be done, it's just more difficult :)

> Of course I can have postfix to skip amavisd for outgoing emails, but
> then I guess I'll lose amavisd's automated whitelisting (the so-called
> 'pen pal' feature).

True. In my case that's not really a problem as we only have amavisd add
a "spam level" header to messages; actually deleting spam is left to the
clients, and most clients that support user-configurable spam policies
and rulesets can do some sort of address whitelisting.

andrea

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Stroller]

On 21/5/2011, at 5:13am, Pandu Poluan wrote:
> ...
> Due to the increase of spam/phishing emails received by my office, I
> decided to explore the idea of implementing a spamfiltering 'frontend'
> in front of my email server.
> 
> Here's how I plan to do it:
> 
> fetchmail (G) --> postfix (G) --> amavisd+spamassassin+database (G)
> --> postfix (G) --> current email back-end (WS) --> clients (W)
> 
> (G) = the single Gentoo server working as mailfilter
> (WS) = mail server on Windows Server
> (W) = various Windows clients (XP and 7)
> 
> I need fetchmail because currently we still use a hosting company, at
> least until August when we host everything on our own. Then, we'll
> drop fetchmail and expose postfix for the world to deliver the mails
> to.

You shouldn't need amavisd / spamassassin, once you're exposing Postfix to the outside world, if you configure it well. 

You should do things like checking that the DNS name matches the helo response given by the server trying to send you mail (this alone filters out a good deal of spam) and be able to use things like DKIM, SPF and even SpamHaus.

http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
http://en.wikipedia.org/wiki/Sender_Policy_Framework
http://www.spamhaus.org/

(SpamHaus says "free for personal use upto x,000 messages per period", but they don't mind business use as long as you're under that limit; still it's cheap, once you've used the free account to prove the service)

Using fetchmail you're unable to reject mail in the same way, so you have to use stuff like amavisd / spamassassin.

Lots of discussion of this on the Postfix mailing list. You should definitely read that for a week or two before deploying.

Stroller.

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Pandu Poluan]

On 2011-05-21, Stroller <stroller@stellar.eclipse.co.uk> wrote:
>
> On 21/5/2011, at 5:13am, Pandu Poluan wrote:
>> ...
>> Due to the increase of spam/phishing emails received by my office, I
>> decided to explore the idea of implementing a spamfiltering 'frontend'
>> in front of my email server.
>>
>> Here's how I plan to do it:
>>
>> fetchmail (G) --> postfix (G) --> amavisd+spamassassin+database (G)
>> --> postfix (G) --> current email back-end (WS) --> clients (W)
>>
>> (G) = the single Gentoo server working as mailfilter
>> (WS) = mail server on Windows Server
>> (W) = various Windows clients (XP and 7)
>>
>> I need fetchmail because currently we still use a hosting company, at
>> least until August when we host everything on our own. Then, we'll
>> drop fetchmail and expose postfix for the world to deliver the mails
>> to.
>
> You shouldn't need amavisd / spamassassin, once you're exposing Postfix to
> the outside world, if you configure it well.
>
> You should do things like checking that the DNS name matches the helo
> response given by the server trying to send you mail (this alone filters out
> a good deal of spam) and be able to use things like DKIM, SPF and even
> SpamHaus.
>
> http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
> http://en.wikipedia.org/wiki/Sender_Policy_Framework
> http://www.spamhaus.org/
>
> (SpamHaus says "free for personal use upto x,000 messages per period", but
> they don't mind business use as long as you're under that limit; still it's
> cheap, once you've used the free account to prove the service)
>
> Using fetchmail you're unable to reject mail in the same way, so you have to
> use stuff like amavisd / spamassassin.
>
> Lots of discussion of this on the Postfix mailing list. You should
> definitely read that for a week or two before deploying.
>
> Stroller.

Well, we've been receiving obvious spams from @yahoo.com, @gmail.com,
and these are valid addresses (apparently people who got phished).
Plus, the Gentoo document I linked earlier also linked to a document
that considers RBLs as... not quite effective.

In addition, if I rely only on DKIM+SPF+RBL, there will be collateral
damage, i.e., false positives. For business reasons, we'd rather have
false negatives (one or two spams got through every week) rather than
false positives. In addition, a cursory check on our clients indicates
that only a few percentage of them implemented SPF. Much less DKIM.

Due to the above reasons, I need a spamfiltering solution that relies
on analyzing the messages themselves.

Rgds,
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Sat, 21 May 2011 16:40:02 +0200, Andrea Conti wrote about Re:
[gentoo-user] RFC: Implementing a spamfiltering frontend:

[snip]
>The problem with having two passes through postfix in the mail routing
>chain is that you either run two separate postfix instances with
>independent configurations or you have to figure out a robust way to
>avoid loops.
>
>It can be done, it's just more difficult :)

Actually, it is quite simple: you just configure a re-injection port in
your original instance of Postfix.  I have been running this
arrangement for some few years now, and I use port 25 for the initial
insertion of an email into Postfix and port 10025 for amavisd to
re-insert the message after spam and virus scans have been completed.
-- 
Regards,

Dave  [RLU #314465]
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
dwnoon@ntlworld.com (David W Noon)
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] RFC: Implementing a spamfiltering frontend

[Stroller]

On 21/5/2011, at 5:14pm, Pandu Poluan wrote:
> ...
> Well, we've been receiving obvious spams from @yahoo.com, @gmail.com,
> and these are valid addresses (apparently people who got phished).

Have you checked they're originating at yahoo / gmail servers? Anyone can spoof a from: address.

> Plus, the Gentoo document I linked earlier also linked to a document
> that considers RBLs as... not quite effective.

I am sceptical of this conclusion, but you certainly shouldn't be relying upon them as if they're a magic bullet. 

> In addition, if I rely only on DKIM+SPF+RBL, there will be collateral
> damage, i.e., false positives.

Only if you choose to reject them on this basis. 

Why don't you greylist messages that fail DKIM/SPF?

> For business reasons, we'd rather have
> false negatives (one or two spams got through every week) rather than
> false positives. In addition, a cursory check on our clients indicates
> that only a few percentage of them implemented SPF. Much less DKIM.
> 
> Due to the above reasons, I need a spamfiltering solution that relies
> on analyzing the messages themselves.

You're not looking at email filtering in a layered, "holistic" manner.

Your answer is "throw spamassassin at the problem, that'll fix it". Personally I've found spamassassin exceedingly poor, if dumbly used in a "naive" manner. 

Since you've done a check on your clients, you already have some hosts you know to permit. Why would you throw away messages from them? If your answer is "because you told me to do DKIM+SPF+RBL" then you're wrong - I just advised you to look at the bigger picture.

Stroller.