[gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Robin H. Johnson]

TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
delivering it. Speak now or hold your peace.

Hi all,

As past long-standing practice, @Gentoo.org system-level mail handling for
incoming mail was officially to tag everything, and delete nothing.

All deletion decisions were left to developers, via procmail/sieve/etc.

This was a good early policy, as Gentoo was a much more reliable host than
email providers a decade ago. This isn't true anymore, with the meteoric rise
and success of gmail.

A LOT of developers forward their mail now, to systems that refuse/temporarily
blacklist the forwarding system because there is a lot of spam. Gmail is
particularly strict in this regard, throttling mail to any recipient from the
forwarding source.

This is particularly acute, because more than 40% of the outgoing mail goes to
Google (the 25% of destinations below is heavily represented because the very
active devs send their mail to google).

This unfortunate combination means that ~40% of mail sits in a backlog for a
long time, and the active devs that use Gmail don't get their mail in a timely
fashion.

Unless there are any major objections, as of May 17th, Infra will start
dropping mail that scores more than 10.0 points in Spamassassin.

If that is successful, I propose to drop the score point by 1 point every month
until it hits a score of 5.0 (so by mid-October, it will be dropping mail that
scores more than 5.0).

Stats on how mail is handled:
-----------------------------
~260 active devs
~180 .forward files

This breaks down to:
~70 procmail users
~10 sieve users
2 users with both forward and procmail
1 maildrop user
~100 devs that send mail outside of @gentoo.org (in their .forward)

I didn't analyze the procmail/sieve/maildrop accounts further.

I did break down the other forwarding destinations by domain:
~50 devs that forward directly to @gmail or @googlemail addresses
~10 devs that have their own domain hosted at gmail/googlemail
~40 devs with some other provider.
0 devs with yahoo, hotmail or msn domains as destinations :-).

As a result, about 25% of dev mail destinations are actually Google.

Amavis stats:
-------------
Here are the amavis summary stats for @gentoo.org incoming mail that was
scanned for content (this happens before exploding to aliases and multiple
recipients, so is a lot lower than you might otherwise expect).

"SPAMMY" in this case is >= 5.5.
     26 May 3 Blocked INFECTED
   1609 May 3 Passed CLEAN
   1564 May 3 Passed SPAMMY
     35 May 4 Blocked INFECTED
   4129 May 4 Passed CLEAN
   2304 May 4 Passed SPAMMY
      2 May 4 Passed UNCHECKED
     42 May 5 Blocked INFECTED
   4458 May 5 Passed CLEAN
   3183 May 5 Passed SPAMMY
      4 May 5 Passed UNCHECKED
     43 May 6 Blocked INFECTED
     10 May 6 Blocked MTA-BLOCKED
   5027 May 6 Passed CLEAN
   3443 May 6 Passed SPAMMY
     47 May 7 Blocked INFECTED
      2 May 7 Blocked MTA-BLOCKED
   4657 May 7 Passed CLEAN
   3119 May 7 Passed SPAMMY
      2 May 7 Passed UNCHECKED
     35 May 8 Blocked INFECTED
   5025 May 8 Passed CLEAN
   2936 May 8 Passed SPAMMY
     21 May 9 Blocked INFECTED
   2497 May 9 Passed CLEAN
   1765 May 9 Passed SPAMMY
     16 May 10 Blocked INFECTED
   2059 May 10 Passed CLEAN
   2033 May 10 Passed SPAMMY

Score analysis of 1 week of incoming mail to amavis:
----------------------------------------------------
~51k unique mails were scored, with a rough breakdown as follows:

~17k < 0.0
~13k 0.0 -  5.0
~7k  5.0 - 10.0
~5k 10.0 - 20.0
~5k 20.0 - 30.0
~3k > 30.0

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Eray Aslan]

On Mon, May 11, 2015 at 04:26:01AM +0000, Robin H. Johnson wrote:
> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
> delivering it. Speak now or hold your peace.

Believe me I understand your pain.  Been there done that.  However,
dropping mail is never a good idea.  You are mucking with the
dependebility of the email.  I would never be able to trust my gentoo
mail if you start dropping spammy mails.  There will always be false
positives.  I suggest:

- Stop forwarding mail.  Have devs pop their mails to whatever account
  they like.  I believe gmail -biggest complainer?- provides this
  option.

- If the above option is not OK for whatever reason, at least let us
  opt-out of the proposed policy of dropping mails provided we do not
  forward our emails.

-- 
Eray

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Tobias Klausmann]

Hi! 

On Mon, 11 May 2015, Eray Aslan wrote:
> On Mon, May 11, 2015 at 04:26:01AM +0000, Robin H. Johnson wrote:
> > TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
> > delivering it. Speak now or hold your peace.
> 
> Believe me I understand your pain.  Been there done that.  However,
> dropping mail is never a good idea.  You are mucking with the
> dependebility of the email.  I would never be able to trust my gentoo
> mail if you start dropping spammy mails.  There will always be false
> positives.  I suggest:
> 
> - Stop forwarding mail.  Have devs pop their mails to whatever account
>   they like.  I believe gmail -biggest complainer?- provides this
>   option.

Big ol' bag of Nope for me. Using POP/IMAP to get mails from
Gentoo servers to where I actually handle mail is a pain in the
rear end. I already let procmail on woodpecker drop all Mails
with a spam score of >=3. I still get shitloads of spam mail that
makes it through. Hence, I'm currently training my own
SpamAssassin for that bit.

That said, I haven't had a false positive from SA in... years?
Maybe even a decade.

> - If the above option is not OK for whatever reason, at least let us
>   opt-out of the proposed policy of dropping mails provided we do not
>   forward our emails.

I'd be fine with that.

Regards,
Tobias
-- 
printk (KERN_INFO "NM256: Congratulations. You're not running Eunice.\n");
	linux-2.6.19/sound/oss/nm256_audio.c

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Tony Vroon]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On 11/05/15 05:26, Robin H. Johnson wrote:
> Unless there are any major objections, as of May 17th, Infra will start
> dropping mail that scores more than 10.0 points in Spamassassin.

This is excellent, as I will then finally be able to forward my Gentoo
alias to the work e-mail server. Like GMail, it is strict because all of
our employees hate spam.

> If that is successful, I propose to drop the score point by 1 point every month
> until it hits a score of 5.0 (so by mid-October, it will be dropping mail that
> scores more than 5.0).

Just speaking as a fellow mail server operator, our automatic throwing
away happens at a score of 8 or above. Between 5 and 8, it is dropped in
a quarantine for sorting by a human.
The amount of false positives is incredibly low, but some of our users
are incredibly vocal. Some of them rely on the human to release that one
mail a month, others are set as spam_lover in amavis and get those
e-mails scoring between 5 & 8 delivered to them.

From a man-power perspective the quarantine may not be realistic, but
perhaps this "spam_lovers" is a good way to allow the opt-out that Eray
is trying to negotiate?

Regards,
Tony V.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 246 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach]

Am Montag, 11. Mai 2015, 04:26:01 schrieb Robin H. Johnson:
> This was a good early policy, as Gentoo was a much more reliable host than
> email providers a decade ago. This isn't true anymore, with the meteoric
> rise
> and success of gmail.
This is not true at all - but email service "reliability" was and is not a 
primarily question of the hosts OS nor some kjind of basic standard 
configuration of a mailer "package" (or ebuild).


> As past long-standing practice, @Gentoo.org system-level mail handling for
> incoming mail was officially to tag everything, and delete nothing.
This is - for a public internet Mailer / MX - a VERY bad option - at least 
mail not fulfilling basic email standards should be blocked (as usual by the 
very most professional level mail services), because it could be (used) 
abusive by thirds.


> A LOT of developers forward their mail now, to systems that
> refuse/temporarily blacklist the forwarding system because there is a lot
> of spam. Gmail is particularly strict in this regard, throttling mail to
> any recipient from the forwarding source.
Gmail is crap (from my opionion) - at least for really professional email 
users or at least users which need a reliable email service (includes the 
possibility to recieve or send out higher frequent emails or emails from more 
"exotic" sources).

We - as a email provider - recognized the development of the rate limiting 
policy by Google as well and it seems that Google is adapting that limits by 
the amount of mail which is send from google users into that "domains" (as far 
as this is correctly locatable for them, because by specs it is not 
really...). This works OK for source mail servers / domains which still have 
"typical" email users and their regular traffic (or what googles thinks about 
- i.e. what some kind of user deletes without reading or is (i.e. false) 
marking as "spam") going to different recipients at google too, but not very 
well for more specialized email systems (like mailing list weighted mail 
systems / MTAs or systems handling some kind of notifications for a larger 
amount of users / customers - (widely) independent from what type of mail they 
transport and how high the part amount of spam within is). 

For mail ISPs there is a contact where they can "complain" or "discuss" about 
limits with a mail server, but i did not know any case wher any answer was 
coming from them (does not mean that google did not recognizes it).

But if someone decied for google as his primarily email provider - he has to 
live with that policies from google, which might work OK for most peoples and 
their "most peoples traffic".


> Unless there are any major objections, as of May 17th, Infra will start
> dropping mail that scores more than 10.0 points in Spamassassin.
> 
> If that is successful, I propose to drop the score point by 1 point every
> month until it hits a score of 5.0 (so by mid-October, it will be dropping
> mail that scores more than 5.0).
This will work (depending form some of your SA setup details and how far you 
use all of the features, channels and possible extensions / third party 
services - i.e. DCC, Razor, Pyzor, "all" the different update channels, Bayes 
- while disabling DNSBLs and doing that still before in your mailer) until you 
go down 5. 

On 6 you typically will still get a lot of spam through (enough to make users 
work to sort it out regularly) while on 5 you definitely will loose ham - even 
if you still use the "usual" DNS based spam blocking lists (would let do this 
the mailer before SA usually because of performance/ressource reasons).

So typical "working" values are between 5 and 6.

But even then you will get spam, as long if you won't loose any ham "from time 
to time" - installing a filter solution like SA alone is just one part of a 
modern and working anti spam gateway with a zero false positive target policy 
(but might be enough to help your moderators here ß). 

Most important tasks are to set up the MTA/MDA far enough to block email which 
are not fulfilling the specs and/or coming from machines which doesnt it - the 
very most of spam should be catched here usually, but is is a dynamic tasks to 
know and/or find out which parts of the standards are important and which 
less, because there are still many used mail systems out which are not 
configured properly, but have a significant user base.

But there are very good reasons why more and more admins of smaller otherwise 
focussed admins decide to route their email traffic over a gateway mail 
service. Setting up a "just working" email system is a simple job today, but 
running a reliable internet email service on a more professional level is a 
real job which takes time and knowledge in maintaining that service - far over 
the setup details of a SMTP daemon (i.e. a proper DNS setup with PTR, DKIM, 
SPF, abuse contacts and so on) which often are out of scope of a typical 
admin.



hth a bit.
cheerioh,


Niels.



-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2200 bytes --]

Hi,

On Mon, 11 May 2015 04:26:01 +0000 Robin H. Johnson wrote:
> As past long-standing practice, @Gentoo.org system-level mail handling for
> incoming mail was officially to tag everything, and delete nothing.
> 
> All deletion decisions were left to developers, via procmail/sieve/etc.
> 
> This was a good early policy, as Gentoo was a much more reliable host than
> email providers a decade ago. This isn't true anymore, with the meteoric rise
> and success of gmail.
> 
> A LOT of developers forward their mail now, to systems that refuse/temporarily
> blacklist the forwarding system because there is a lot of spam. Gmail is
> particularly strict in this regard, throttling mail to any recipient from the
> forwarding source.

Unconditional adjustment of free software infrastructure for very
questionable rules of proprietary product is a very bad idea.

> This is particularly acute, because more than 40% of the outgoing mail goes to
> Google (the 25% of destinations below is heavily represented because the very
> active devs send their mail to google).
> 
> This unfortunate combination means that ~40% of mail sits in a backlog for a
> long time, and the active devs that use Gmail don't get their mail in a timely
> fashion.

Make this dropping optional: if devs are using gmail and really need
that filtering, they can opt-in. Left it opt-out for other devs.

Mail filtering is a minefield: too much spam is bad, loosing
even single important e-mail due to over restrictive filter is even
worse.

I've had enough with over restrictive mail servers, e.g. blocking
entire countries and ip ranges. I don't want to see Gentoo going
that way too.

> Unless there are any major objections, as of May 17th, Infra will start
> dropping mail that scores more than 10.0 points in Spamassassin.
> 
> If that is successful, I propose to drop the score point by 1 point every month
> until it hits a score of 5.0 (so by mid-October, it will be dropping mail that
> scores more than 5.0).

Why so much focus on spamassassin? Why not to use (perhaps in
addition) more elegant technologies as the double grey listing?

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Am Montag, 11. Mai 2015, 15:39:13 schrieb Andrew Savchenko:
> Mail filtering is a minefield: too much spam is bad, loosing
> even single important e-mail due to over restrictive filter is even
> worse.
This is true, as far as you go over standard compliance checks and unserstand 
standard violating (absuive) mail as "spam" in any case - it (not at least) 
depends from what you DEFINE as "Spam" (or abusive Email).

Each working internet mails ystem (working means: is able to send email to the 
very most of recievers) has to filter out some kind of mail traffic - at least 
proven hardly abusive or what the majority of mail server admins understands 
as "abusive" (otherwise your own users mails get blocked from others). This 
means that you WILL block emails from regular users too, unable to send a 
"correct" email.


hth a bit,


Niels.

-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 




[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Charles Nérot]

Hello,

Lot of thing are done for fighting spam : dnssec, dane, spf, dkim,
dmarc... All of this for "trusting real sender".
Some of them break  smtp built in fonctionnality : spf break forwarding [1].

If you beleive in spf (gentoo.org have an spf dns entry) , two ways need
to be looked at :
- fixing real sender with SRS [1].
- stop forwarding mail and do POP (gmail can do it) or IMAP from your
favorite (web)mail client.

Dmarc dns entry with report activated can help you understand why google
blacklist you.

[1] http://www.openspf.org/SRS

Regards,
Charles Nérot

Le 11/05/2015 06:26, Robin H. Johnson a écrit :
> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
> delivering it. Speak now or hold your peace.
>
> Hi all,
>
> As past long-standing practice, @Gentoo.org system-level mail handling for
> incoming mail was officially to tag everything, and delete nothing.
>
> All deletion decisions were left to developers, via procmail/sieve/etc.
>
> This was a good early policy, as Gentoo was a much more reliable host than
> email providers a decade ago. This isn't true anymore, with the meteoric rise
> and success of gmail.
>
> A LOT of developers forward their mail now, to systems that refuse/temporarily
> blacklist the forwarding system because there is a lot of spam. Gmail is
> particularly strict in this regard, throttling mail to any recipient from the
> forwarding source.
>
> This is particularly acute, because more than 40% of the outgoing mail goes to
> Google (the 25% of destinations below is heavily represented because the very
> active devs send their mail to google).
>
> This unfortunate combination means that ~40% of mail sits in a backlog for a
> long time, and the active devs that use Gmail don't get their mail in a timely
> fashion.
>
> Unless there are any major objections, as of May 17th, Infra will start
> dropping mail that scores more than 10.0 points in Spamassassin.
>
> If that is successful, I propose to drop the score point by 1 point every month
> until it hits a score of 5.0 (so by mid-October, it will be dropping mail that
> scores more than 5.0).
>
> Stats on how mail is handled:
> -----------------------------
> ~260 active devs
> ~180 .forward files
>
> This breaks down to:
> ~70 procmail users
> ~10 sieve users
> 2 users with both forward and procmail
> 1 maildrop user
> ~100 devs that send mail outside of @gentoo.org (in their .forward)
>
> I didn't analyze the procmail/sieve/maildrop accounts further.
>
> I did break down the other forwarding destinations by domain:
> ~50 devs that forward directly to @gmail or @googlemail addresses
> ~10 devs that have their own domain hosted at gmail/googlemail
> ~40 devs with some other provider.
> 0 devs with yahoo, hotmail or msn domains as destinations :-).
>
> As a result, about 25% of dev mail destinations are actually Google.
>
> Amavis stats:
> -------------
> Here are the amavis summary stats for @gentoo.org incoming mail that was
> scanned for content (this happens before exploding to aliases and multiple
> recipients, so is a lot lower than you might otherwise expect).
>
> "SPAMMY" in this case is >= 5.5.
>      26 May 3 Blocked INFECTED
>    1609 May 3 Passed CLEAN
>    1564 May 3 Passed SPAMMY
>      35 May 4 Blocked INFECTED
>    4129 May 4 Passed CLEAN
>    2304 May 4 Passed SPAMMY
>       2 May 4 Passed UNCHECKED
>      42 May 5 Blocked INFECTED
>    4458 May 5 Passed CLEAN
>    3183 May 5 Passed SPAMMY
>       4 May 5 Passed UNCHECKED
>      43 May 6 Blocked INFECTED
>      10 May 6 Blocked MTA-BLOCKED
>    5027 May 6 Passed CLEAN
>    3443 May 6 Passed SPAMMY
>      47 May 7 Blocked INFECTED
>       2 May 7 Blocked MTA-BLOCKED
>    4657 May 7 Passed CLEAN
>    3119 May 7 Passed SPAMMY
>       2 May 7 Passed UNCHECKED
>      35 May 8 Blocked INFECTED
>    5025 May 8 Passed CLEAN
>    2936 May 8 Passed SPAMMY
>      21 May 9 Blocked INFECTED
>    2497 May 9 Passed CLEAN
>    1765 May 9 Passed SPAMMY
>      16 May 10 Blocked INFECTED
>    2059 May 10 Passed CLEAN
>    2033 May 10 Passed SPAMMY
>
> Score analysis of 1 week of incoming mail to amavis:
> ----------------------------------------------------
> ~51k unique mails were scored, with a rough breakdown as follows:
>
> ~17k < 0.0
> ~13k 0.0 -  5.0
> ~7k  5.0 - 10.0
> ~5k 10.0 - 20.0
> ~5k 20.0 - 30.0
> ~3k > 30.0
>

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

Sorry to shoot and run, but I think you're trying to tackle this
problem in the wrong way. The problem isn't to drop the mail. The
solution is to change email hosting providers. As a non-profit I
believe Google hosted apps would be an option (free). Then it would be
possible to simply leverage that service and not have to worry about
the forwarding. (You'd maybe save time and get (great?) spam filtering
in the process)

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Mon, May 11, 2015 at 9:37 AM, C Bergström <cbergstrom@pathscale.com> wrote:
> Sorry to shoot and run, but I think you're trying to tackle this
> problem in the wrong way. The problem isn't to drop the mail. The
> solution is to change email hosting providers. As a non-profit I
> believe Google hosted apps would be an option (free).

In general we try to stick to our social contract, and that means
trying to avoid depending on proprietary technologies such as gmail.

Now, I could see just using a FOSS-based IMAP/SMTP/POP provider,
perhaps which allows things like forwarding and such, which allows us
to have a copy of all our configuration and such in case we want to
migrate.  I'm not super-familiar with the wordpress.com model but
something like that also seems reasonable - we leverage donations of
hosting services but we aren't bound to anything proprietary and have
the ability to migrate off.

I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
is), and ditto for Google docs (or whatever the latest branding for
that is). There is nothing magical about cloud-based services any more
than there is anything magical about letting somebody else host your
website.  The key is to ensure that the technologies are open so that
you aren't bound to a single provider.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

What I'm describing is not "gmail" - it's everything that gmail has
and offers, but @gentoo.org domain. I'm using it right now in fact.

You get the web interface, IMAP, POP, 2 token authentication (if you
want to enabled it) and lots of other things. etc etc

It used to be free, but now google charges for it with an exception
for non-profits. The admin could test it and see if it provides all
the features you need. If you do SSO it may even be possible to
integrate with existing infrastructure.. blah blah.. I'd highly
recommend someone who actually has to manage this take a look.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Mon, May 11, 2015 at 10:44 AM, C Bergström <cbergstrom@pathscale.com> wrote:
> What I'm describing is not "gmail" - it's everything that gmail has
> and offers, but @gentoo.org domain. I'm using it right now in fact.
>
> You get the web interface, IMAP, POP, 2 token authentication (if you
> want to enabled it) and lots of other things. etc etc

How about the source code?

>
> It used to be free, but now google charges for it with an exception
> for non-profits.

The social contract isn't about free-of-cost.  In fact, Gentoo pays
for a number of services (often below commercial rates, but not
everybody can afford to donate 100% of what we need).  We've even paid
for a bug bounty on one occasion.  The social contract is about
free-as-in-freedom.  We don't depend on proprietary services as much
as possible.

We even have debates over the use of github, since the pull request
side isn't really FOSS.  It is tolerated mainly because we have FOSS
alternatives as well, and bugzilla is still the primary bug
tracker/etc.  To the extent that github is just used as a hosting
provider for git it is completely compatible with the social contract,
and would be so even if we were paying for it.

All that said, being non-profit we still try to use donations of
services anytime we can.  Our mirror network is probably the biggest
example of this - we have an insane amount of mirror bandwidth and
there is no way an org of our size could afford to pay for it on our
own.  Next time you do an emerge --sync take a look at the
hostnames/MOTDs/etc and be sure to appreciate them in some way.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

On Mon, May 11, 2015 at 9:59 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, May 11, 2015 at 10:44 AM, C Bergström <cbergstrom@pathscale.com> wrote:
>> What I'm describing is not "gmail" - it's everything that gmail has
>> and offers, but @gentoo.org domain. I'm using it right now in fact.
>>
>> You get the web interface, IMAP, POP, 2 token authentication (if you
>> want to enabled it) and lots of other things. etc etc
>
> How about the source code?

Do you have the source for github?

>
>>
>> It used to be free, but now google charges for it with an exception
>> for non-profits.
>
> The social contract isn't about free-of-cost.  In fact, Gentoo pays
> for a number of services (often below commercial rates, but not
> everybody can afford to donate 100% of what we need).  We've even paid
> for a bug bounty on one occasion.  The social contract is about
> free-as-in-freedom.  We don't depend on proprietary services as much
> as possible.
>
> We even have debates over the use of github, since the pull request
> side isn't really FOSS.  It is tolerated mainly because we have FOSS
> alternatives as well, and bugzilla is still the primary bug
> tracker/etc.  To the extent that github is just used as a hosting
> provider for git it is completely compatible with the social contract,
> and would be so even if we were paying for it.

There are "free" alternatives and this is the exact same thing as
github. IMAP and POP are comparable to git as google hosted apps is
comparable to github. There's a line between being passionate and
ignoring a sensible good alternative. I can't say where to draw that
line, but imho I hope pragmatic people will take a look instead of
just dismissing it.

Oh and btw - the whole problem comes because people are forwarding to
gmail. Is that open source? It's clear a large number of people
already use and depend on the exact same service I'm suggesting. How
on earth could those same people object... (I don't see the open
source communit up in arms over yahoo mail and gmail..)

/* I'm just trying to level the conversation in terms of "social
contract" and what people generally find acceptable */
Do you own a phone that connects to this email? Android, iOS.. etc
aren't "open source", but somehow we survive..

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Alexis Ballier]

On Mon, 11 May 2015 22:21:09 +0700
C Bergström <cbergstrom@pathscale.com> wrote:

> On Mon, May 11, 2015 at 9:59 PM, Rich Freeman <rich0@gentoo.org>
> wrote:
> > On Mon, May 11, 2015 at 10:44 AM, C Bergström
> > <cbergstrom@pathscale.com> wrote:
> >> What I'm describing is not "gmail" - it's everything that gmail has
> >> and offers, but @gentoo.org domain. I'm using it right now in fact.
> >>
> >> You get the web interface, IMAP, POP, 2 token authentication (if
> >> you want to enabled it) and lots of other things. etc etc
> >
> > How about the source code?
> 
> Do you have the source for github?
>

You should probably think about the difference between public code being
mirrored at github and giving some big company access to private
emails.

Alexis.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Matthew Thode]

[-- Attachment #1: Type: text/plain, Size: 2619 bytes --]

On 05/11/2015 10:21 AM, C Bergström wrote:
> On Mon, May 11, 2015 at 9:59 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> On Mon, May 11, 2015 at 10:44 AM, C Bergström <cbergstrom@pathscale.com> wrote:
>>> What I'm describing is not "gmail" - it's everything that gmail has
>>> and offers, but @gentoo.org domain. I'm using it right now in fact.
>>>
>>> You get the web interface, IMAP, POP, 2 token authentication (if you
>>> want to enabled it) and lots of other things. etc etc
>>
>> How about the source code?
> 
> Do you have the source for github?
No, but we get flack for that all the time, I'd personally like to see
us use bitbucket as they have a more opensource and doc'd stack.
> 
>>
>>>
>>> It used to be free, but now google charges for it with an exception
>>> for non-profits.
>>
>> The social contract isn't about free-of-cost.  In fact, Gentoo pays
>> for a number of services (often below commercial rates, but not
>> everybody can afford to donate 100% of what we need).  We've even paid
>> for a bug bounty on one occasion.  The social contract is about
>> free-as-in-freedom.  We don't depend on proprietary services as much
>> as possible.
>>
>> We even have debates over the use of github, since the pull request
>> side isn't really FOSS.  It is tolerated mainly because we have FOSS
>> alternatives as well, and bugzilla is still the primary bug
>> tracker/etc.  To the extent that github is just used as a hosting
>> provider for git it is completely compatible with the social contract,
>> and would be so even if we were paying for it.
> 
> There are "free" alternatives and this is the exact same thing as
> github. IMAP and POP are comparable to git as google hosted apps is
> comparable to github. There's a line between being passionate and
> ignoring a sensible good alternative. I can't say where to draw that
> line, but imho I hope pragmatic people will take a look instead of
> just dismissing it.
> 
> Oh and btw - the whole problem comes because people are forwarding to
> gmail. Is that open source? It's clear a large number of people
> already use and depend on the exact same service I'm suggesting. How
> on earth could those same people object... (I don't see the open
> source communit up in arms over yahoo mail and gmail..)
> 
> /* I'm just trying to level the conversation in terms of "social
> contract" and what people generally find acceptable */
> Do you own a phone that connects to this email? Android, iOS.. etc
> aren't "open source", but somehow we survive..
> 


-- 
-- Matthew Thode (prometheanfire)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

On Mon, 11 May 2015 18:17:10 +0200
Alexis Ballier <aballier@gentoo.org> wrote:
> You should probably think about the difference between public code
> being mirrored at github and giving some big company access to private
> emails.

Like your phone company, ISP, and national intelligence agencies?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

Look at the forwarding which is already happening. They are already
giving that big company the emails. That big company gets a copy of
every email which is posted publicly already.

Are you concerned about their privacy policy? Are you concerned about
them complying to a government demand or ads.. What's your exact
concern here..

Keep in mind that github has access to your data and their EULA
probably allows them to run whatever analysis they want. A that the
high level it's essentially the same thing. The difference is that
github isn't smart enough to tie revenue into this. (Such as
displaying ads to C/C++ devs)

"we" (most technically savvy people who own a smart phone) can't avoid
these big companies. It's just a fact of life. For a small
organization like gentoo - it should be respected when time saving
choices are made that allow everyone to just get things done.

I hope "social contract" doesn't mean zealot and instead means someone
who is pragmatic... What's the real concern here.. pragmatically.
(honest question)

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Alexis Ballier]

On Mon, 11 May 2015 17:20:01 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 11 May 2015 18:17:10 +0200
> Alexis Ballier <aballier@gentoo.org> wrote:
> > You should probably think about the difference between public code
> > being mirrored at github and giving some big company access to
> > private emails.
> 
> Like your phone company, ISP, and national intelligence agencies?

I think you know the answer for those...

And even with clear text emails going through them, unless
they've managed to secretly obtain decades of cryptography advances,
they can't know e.g. what I do actually read and even less how much time
I spend reading it (and probably dozens of others statistics I can't
even imagine), which correlation is actually much more valuable than
the raw data itself.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

Dnia 2015-05-11, o godz. 17:20:01
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> napisał(a):

> On Mon, 11 May 2015 18:17:10 +0200
> Alexis Ballier <aballier@gentoo.org> wrote:
> > You should probably think about the difference between public code
> > being mirrored at github and giving some big company access to private
> > emails.
> 
> Like your phone company, ISP, and national intelligence agencies?

Don't forget all the companies involved in hosting the Gentoo services.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Mon, May 11, 2015 at 11:21 AM, C Bergström <cbergstrom@pathscale.com> wrote:
> On Mon, May 11, 2015 at 9:59 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> On Mon, May 11, 2015 at 10:44 AM, C Bergström <cbergstrom@pathscale.com> wrote:
>>> What I'm describing is not "gmail" - it's everything that gmail has
>>> and offers, but @gentoo.org domain. I'm using it right now in fact.
>>>
>>> You get the web interface, IMAP, POP, 2 token authentication (if you
>>> want to enabled it) and lots of other things. etc etc
>>
>> How about the source code?
>
> Do you have the source for github?
>

That would be why I point out that we have debates over its use.
Right now we're accepting it mainly because it is an alternative.  I
suspect there would be more concern if it were proposed that we move
to it exclusively.

>
> There are "free" alternatives and this is the exact same thing as
> github. IMAP and POP are comparable to git as google hosted apps is
> comparable to github.

I think that is actually your best line of argument.  They're just a
black box that accepts SMTP and makes available POP, and that is all
Gentoo uses.  They just happen to offer an MUA on the side, which g.o
devs/staff can use at their discretion.

I'm not convinced it is the way to go, all the same.

> There's a line between being passionate and
> ignoring a sensible good alternative. I can't say where to draw that
> line, but imho I hope pragmatic people will take a look instead of
> just dismissing it.
>

I have nothing against Gmail and I'm composing this email using it.
Even so, unless we were simply unable to host our own mail, I'm not
sure I'd advocate moving g.o over to Google apps.

That doesn't mean I'm not willing to be pragmatic.  I just think that
making a change like this really should be done only out of necessity.
There are other Gentoo services that I'd sooner see go than email.

> Oh and btw - the whole problem comes because people are forwarding to
> gmail. Is that open source? It's clear a large number of people
> already use and depend on the exact same service I'm suggesting. How
> on earth could those same people object... (I don't see the open
> source communit up in arms over yahoo mail and gmail..)

I can use Gmail personally and still object to forcing everybody else
to use it.  I don't see that as hypocrisy at all.

> Do you own a phone that connects to this email? Android, iOS.. etc
> aren't "open source", but somehow we survive..

Android is more or less open-source.  There are some blobs in it, but
not all that much more than you'd find on a typical PC running Gentoo.
But again, Gentoo doesn't provide cell phones or require that anybody
own one.  If somebody wants to read dev.g.o on their iPhone, I don't
have a problem with that.  If we were to force anybody who wanted to
subscribe to dev.g.o to own an iPhone I would have a problem with
that.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

On Mon, May 11, 2015 at 11:55 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, May 11, 2015 at 11:21 AM, C Bergström <cbergstrom@pathscale.com> wrote:
>> On Mon, May 11, 2015 at 9:59 PM, Rich Freeman <rich0@gentoo.org> wrote:
>>> On Mon, May 11, 2015 at 10:44 AM, C Bergström <cbergstrom@pathscale.com> wrote:
>>>> What I'm describing is not "gmail" - it's everything that gmail has
>>>> and offers, but @gentoo.org domain. I'm using it right now in fact.
>>>>
>>>> You get the web interface, IMAP, POP, 2 token authentication (if you
>>>> want to enabled it) and lots of other things. etc etc
>>>
>>> How about the source code?
>>
>> Do you have the source for github?
>>
>
> That would be why I point out that we have debates over its use.
> Right now we're accepting it mainly because it is an alternative.  I
> suspect there would be more concern if it were proposed that we move
> to it exclusively.
>
>>
>> There are "free" alternatives and this is the exact same thing as
>> github. IMAP and POP are comparable to git as google hosted apps is
>> comparable to github.
>
> I think that is actually your best line of argument.  They're just a
> black box that accepts SMTP and makes available POP, and that is all
> Gentoo uses.  They just happen to offer an MUA on the side, which g.o
> devs/staff can use at their discretion.
>
> I'm not convinced it is the way to go, all the same.
>
>> There's a line between being passionate and
>> ignoring a sensible good alternative. I can't say where to draw that
>> line, but imho I hope pragmatic people will take a look instead of
>> just dismissing it.
>>
>
> I have nothing against Gmail and I'm composing this email using it.
> Even so, unless we were simply unable to host our own mail, I'm not
> sure I'd advocate moving g.o over to Google apps.
>
> That doesn't mean I'm not willing to be pragmatic.  I just think that
> making a change like this really should be done only out of necessity.
> There are other Gentoo services that I'd sooner see go than email.
>
>> Oh and btw - the whole problem comes because people are forwarding to
>> gmail. Is that open source? It's clear a large number of people
>> already use and depend on the exact same service I'm suggesting. How
>> on earth could those same people object... (I don't see the open
>> source communit up in arms over yahoo mail and gmail..)
>
> I can use Gmail personally and still object to forcing everybody else
> to use it.  I don't see that as hypocrisy at all.
>
>> Do you own a phone that connects to this email? Android, iOS.. etc
>> aren't "open source", but somehow we survive..
>
> Android is more or less open-source.  There are some blobs in it, but
> not all that much more than you'd find on a typical PC running Gentoo.
> But again, Gentoo doesn't provide cell phones or require that anybody
> own one.  If somebody wants to read dev.g.o on their iPhone, I don't
> have a problem with that.  If we were to force anybody who wanted to
> subscribe to dev.g.o to own an iPhone I would have a problem with
> that.

Android is much less open than people think or want to believe.
Entirely besides that point - I hope this could be proposed and at
least looked at as an option. I have ran a postfix setup with
graylisting and spam filtering. It's quite a bit of maintenance to
ensure it's done in a high quality manner. The time/cost/energy in
that vs something which "just works" is hard to pass up. I realize
philosophies and religions differ - I respect your view and hope a
good solution is found in the end. If not now - I suspect at some
point gentoo will relinquish their email to a service provider..

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Michael Orlitzky]

On 05/11/2015 03:29 AM, Eray Aslan wrote:
> On Mon, May 11, 2015 at 04:26:01AM +0000, Robin H. Johnson wrote:
>> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
>> delivering it. Speak now or hold your peace.
> 
> Believe me I understand your pain.  Been there done that.  However,
> dropping mail is never a good idea.  You are mucking with the
> dependebility of the email.

Agreed.

Is there some reason a pre-queue filter (with amavisd-new) wouldn't
work? Then we could reject the spammy messages (at SMTP time) instead of
silently dropping them.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Kristian Fiskerstrand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 05/11/2015 09:31 PM, Michael Orlitzky wrote:
> On 05/11/2015 03:29 AM, Eray Aslan wrote:
>> On Mon, May 11, 2015 at 04:26:01AM +0000, Robin H. Johnson
>> wrote:
>>> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail
>>> instead of delivering it. Speak now or hold your peace.
>> 
>> Believe me I understand your pain.  Been there done that.
>> However, dropping mail is never a good idea.  You are mucking
>> with the dependebility of the email.
> 
> Agreed.
> 
> Is there some reason a pre-queue filter (with amavisd-new)
> wouldn't work? Then we could reject the spammy messages (at SMTP
> time) instead of silently dropping them.
> 
> 

Could it be an alternative to move the messages flagged as spam into
an own folder that isn't forwarded? at least that means it doesn't
impact operations for those using it locally and the mail is still
around, if a webmail interface or something was used it could be
accessed through that for the forwarding users.


- -- 
Kristian Fiskerstrand
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJVUQSHAAoJEP7VAChXwav6nYUH/RI8LsL3/xEFgZXXlVy0ULXI
vprolO9hApBgn5gAsUld9+LXCNe/oHDlAVd7TE2AnfHmsIG7yHyC2oZyl76X1xk9
/8bztTuJRhXuKo3jt3UO+Nx8HP/BfVv0CdyJ4RrunPt5qG8o6DpNTkD8Y2KPhdFU
clnqzllt/2vk15eB4IFjL9U/s8ZMeCI+S36tFPpS8XBXQEbwhMxZ127XyoaXKwBq
q+eH4uTZw1piggOo/JeiEpDUPqn27Jvoth7cYzyS0vnfUkPKpHbYglnT7zz1fNZ+
+FfGyazMSzPS9hICYyTvNyhS4Ya8o25Ep0xoEZC4TUSg/4JMrUqwNUBaZNzxwaM=
=HwhU
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Michael Orlitzky]

On 05/11/2015 03:35 PM, Kristian Fiskerstrand wrote:
> 
> Could it be an alternative to move the messages flagged as spam into
> an own folder that isn't forwarded? at least that means it doesn't
> impact operations for those using it locally and the mail is still
> around, if a webmail interface or something was used it could be
> accessed through that for the forwarding users.
> 

Yes, and depending on the sieve implementation, we can make sure the
users (ourselves, in this case) don't override the rule and mess things up.

The first thing to do would be to change the documentation: instead of
using a ~/.forward file to forward your @g.o mail, you would use sieve:

  require ["copy"];
  redirect :copy "wherever@example.org";

Then the global sieve file (run first) could do,

  require "fileinto";
  if header :contains "X-Spam-Flag" "YES" {
    fileinto "Junk";
    stop;
  }

which would stop processing, so that the "redirect" above never gets
executed.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Robin H. Johnson]

On Mon, May 11, 2015 at 03:31:51PM -0400, Michael Orlitzky wrote:
> On 05/11/2015 03:29 AM, Eray Aslan wrote:
> > On Mon, May 11, 2015 at 04:26:01AM +0000, Robin H. Johnson wrote:
> >> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail instead of
> >> delivering it. Speak now or hold your peace.
> > 
> > Believe me I understand your pain.  Been there done that.  However,
> > dropping mail is never a good idea.  You are mucking with the
> > dependebility of the email.
> Is there some reason a pre-queue filter (with amavisd-new) wouldn't
> work? Then we could reject the spammy messages (at SMTP time) instead of
> silently dropping them.
By drop, I will clarify that they should ideally be rejected at SMTP
time, not silently dropped.

amavis settings for this in theory are:
$sa_kill_level_deflt = 20.0;
$sa_dsn_cutoff_level = 20.0;
$sa_crediblefrom_dsn_cutoff_level = 20.0;
$final_virus_destiny      = D_REJECT;
$final_banned_destiny     = D_REJECT;
$final_spam_destiny       = D_REJECT; 

(The other choices are D_PASS, D_BOUNCE, D_DISCARD)
D_REJECT is supposed to just reply to the SMTP error.

It's doing that, but it's ALSO sending a NDN, despite amavis settings to
the contrary.

Here's a quick test, with D_REJECT that show the NDN being sent (and rejected
since that address was probably faked).

May 11 20:00:45 woodpecker postfix/smtpd[21896]: E83DF34098C: client=unknown[183.93.114.52]
May 11 20:00:46 woodpecker postfix/cleanup[21836]: E83DF34098C: message-id=<>
May 11 20:00:46 woodpecker postfix/qmgr[21745]: E83DF34098C: from=<mldmh@bjchwa.com>, size=5678, nrcpt=1 (queue active)
May 11 20:00:53 woodpecker amavis[21935]: (21935-01) Blocked SPAM {RejectedInbound}, [183.93.114.52]:4758 [183.93.114.52] <mldmh@bjchwa.com> -> <$DEVNAME@gentoo.org>, Queue-ID: E83DF34098C, mail_id: 6k-hfYzAtEKp, Hits: 26.004, size: 5678, 6812 ms
May 11 20:00:53 woodpecker postfix/bounce[21847]: E83DF34098C: sender non-delivery notification: 7ECE534098D
May 11 20:00:53 woodpecker postfix/cleanup[21754]: 7ECE534098D: message-id=<20150511200053.7ECE534098D@smtp.gentoo.org>
May 11 20:00:53 woodpecker postfix/qmgr[21745]: 7ECE534098D: from=<>, size=7622, nrcpt=1 (queue active)
May 11 20:00:53 woodpecker postfix/qmgr[21745]: E83DF34098C: removed
May 11 20:00:53 woodpecker postfix/smtp[21837]: E83DF34098C: to=<$DEVNAME@gentoo.org>, relay=127.0.0.1[127.0.0.1]:10024, delay=12, delays=4.8/0/0.01/6.8, dsn=5.7.0, status=bounced (host 127.0.0.1[127.0.0.1] said: 554 5.7.0 Reject, id=21935-01 - spam (in reply to end of DATA command))
May 11 20:01:25 woodpecker postfix/qmgr[21745]: 7ECE534098D: removed
May 11 20:01:25 woodpecker postfix/smtp[21773]: 7ECE534098D: to=<mldmh@bjchwa.com>, relay=mxbiz1.qq.com[184.105.206.87]:25, delay=32, delays=0/0/30/1, dsn=5.0.0, status=bounced (host mxbiz1.qq.com[184.105.206.87] said: 550 Mail content denied. http://service.mail.qq.com/cgi-bin/help?subtype=1&&id=20022&&no=1000726 (in reply to end of DATA command))

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Robin H. Johnson]

On Mon, May 11, 2015 at 03:39:13PM +0300, Andrew Savchenko wrote:
> Unconditional adjustment of free software infrastructure for very
> questionable rules of proprietary product is a very bad idea.
It's an ecosystem. If we do nothing, we continue to penalize all
developers who forward their mail to Google, as well as ANY users who
would get mail from us sent to their Google accounts (forums
notifications, bugzilla mail).

> > This is particularly acute, because more than 40% of the outgoing mail goes to
> > Google (the 25% of destinations below is heavily represented because the very
> > active devs send their mail to google).
> > 
> > This unfortunate combination means that ~40% of mail sits in a backlog for a
> > long time, and the active devs that use Gmail don't get their mail in a timely
> > fashion.
> Make this dropping optional: if devs are using gmail and really need
> that filtering, they can opt-in. Left it opt-out for other devs.
We ALREADY have the .permissive file, and have for many many years.
Documented here:
https://wiki.gentoo.org/wiki/Project:Infrastructure/Developer_E-Mail#How_can_I_exempt_myself_from_Sender_Address_Verification.3F

But, one of the problems with it is mail aliases. It cannot be set for
any of the aliases, because the exploding of recipients is only done
after receiving the mail.

> Mail filtering is a minefield: too much spam is bad, loosing
> even single important e-mail due to over restrictive filter is even
> worse.
> 
> I've had enough with over restrictive mail servers, e.g. blocking
> entire countries and ip ranges. I don't want to see Gentoo going
> that way too.
> 
> > Unless there are any major objections, as of May 17th, Infra will start
> > dropping mail that scores more than 10.0 points in Spamassassin.
> > 
> > If that is successful, I propose to drop the score point by 1 point every month
> > until it hits a score of 5.0 (so by mid-October, it will be dropping mail that
> > scores more than 5.0).
> Why so much focus on spamassassin? Why not to use (perhaps in
> addition) more elegant technologies as the double grey listing?
How about asking what Infra is using before that?
amavis
spamassassin
greylisting
sender address verification
SPF (* with a permissive policy)
DNSSEC
Pyzor
Razor2
DCC
TextCat
DKIM verifiation
OCR** (was mostly a failure)

SA channels in use are:
updates.spamassassin.org
sought.rules.yerp.org
malware.org (* presently disabled, sa-compile takes 1hr+ on newer spamassassin)
SARE was previously in use, but hasn't been maintained for years.

You can see more of the details in bug #539420 as well.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Robin H. Johnson]

On Mon, May 11, 2015 at 12:09:08PM +0200, Niels Dettenbach wrote:
> > As past long-standing practice, @Gentoo.org system-level mail handling for
> > incoming mail was officially to tag everything, and delete nothing.
> This is - for a public internet Mailer / MX - a VERY bad option - at least 
> mail not fulfilling basic email standards should be blocked (as usual by the 
> very most professional level mail services), because it could be (used) 
> abusive by thirds.
There are people that still accept mail that violates standards?
My above statement is for mail that we ACCEPTED. If it violates
standards, it's already denied at SMTP time.

smtpd_restriction_classes = restrictive,permissive
restrictive =
    reject_invalid_hostname
    reject_non_fqdn_hostname
    reject_non_fqdn_recipient
    reject_non_fqdn_sender
    reject_unknown_sender_domain
    reject_unknown_recipient_domain
    check_sender_mx_access cidr:/etc/postfix/bogus_mx_records
    check_sender_access pcre:/etc/postfix/sender_access_control.pcre
    check_sender_access pcre:/etc/postfix/sender_access_control-aliases.pcre
    check_helo_access pcre:/etc/postfix/helo_checks
    reject_unverified_sender
    check_client_access cidr:/etc/postfix/filter.cidr
    permit
permissive =
    permit

> > Unless there are any major objections, as of May 17th, Infra will start
> > dropping mail that scores more than 10.0 points in Spamassassin.
> > 
> > If that is successful, I propose to drop the score point by 1 point every
> > month until it hits a score of 5.0 (so by mid-October, it will be dropping
> > mail that scores more than 5.0).
> This will work (depending form some of your SA setup details and how far you 
> use all of the features, channels and possible extensions / third party 
> services - i.e. DCC, Razor, Pyzor, "all" the different update channels, Bayes 
> - while disabling DNSBLs and doing that still before in your mailer) until you 
> go down 5. 
See my other response, we've got pretty much all of the things going already.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Michael Orlitzky]

On 05/11/2015 04:08 PM, Robin H. Johnson wrote:
> By drop, I will clarify that they should ideally be rejected at SMTP
> time, not silently dropped.
> 

I believe those logs show a rejection after the message has been
accepted initially (if I'm wrong, you can ignore the rest of this). This
is better in a way, but causes another problem.

Obviously it's better than a silent discard in that, in the case of a
false positive, the sender will be notified of the mistake. But, it
creates backscatter[1] which can get us blacklisted or be used as part
of a DDoS. It's also just plain rude to the innocent people we're
bombarding with rejections (for mail they never sent).

That happens because we,

  1. Accept the message (postfix)
  2. Queue the message (postfix)
  3. Client disconnects (postfix)
  4. Scan the message (amavis)
  5. Decide to reject the message (amavis)
  6. Alert the sender (postfix)

This saves resources because there's a queue waiting to get into amavis,
and scanning can take a while. As long as we scan at a decent *average*
speed, the queue will eventually empty.

But at step #4, the original client is long gone, so you can't just feed
him a 5xx error. There's another "pre-queue" mode for amavis which lets
us reject right away. If you configure e.g.

  smtpd_proxy_filter = localhost:10024   # amavis goes here

in Postfix's main.cf, then amavis will scan the message as its being
received. Then the D_REJECT will actually just say "5xx: get lost" to
the client while he's still sending the message. Real senders will get
an NDR from their mail server. And if a spammer uses a forged address,
it doesn't matter -- the spammer gets the rejection, not the poor guy
whose email he used. It looks something like,

  1. Client connects
  2. Postfix proxies the message through amavis
  3. Amavis scans the message and reports its verdict to postfix
  4. Postfix reports the good/bad news to the client
  5. Message is queued (or not)
  6. Client disconnects

The downside is that the "pre-queue" mode needs a lot more resources.
When there's no "going to amavis" queue, you need enough resources
(amavis processes, memory...) around to be able to scan all of your
incoming mail at once. Otherwise you'll run out of resources and clients
will start seeing 4xx errors (this isn't that bad if it only happens
under rare circumstances).

FWIW we do this for a few thousand accounts on a Proliant DL360 G5. I'm
happy to share any of the configuration.



[1] http://en.wikipedia.org/wiki/Backscatter_%28email%29

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Robin H. Johnson]

On Mon, May 11, 2015 at 03:27:12PM +0200, Charles Nérot wrote:
> Lot of thing are done for fighting spam : dnssec, dane, spf, dkim,
> dmarc... All of this for "trusting real sender".
> Some of them break  smtp built in fonctionnality : spf break forwarding [1].
DANE does nothing for spam, there are spammers that pass DNSSEC, DANE, DKIM,
SPF. DMARC breaks mailing lists badly for domains with reject as their policy [1].

> If you beleive in spf (gentoo.org have an spf dns entry) , two ways need
> to be looked at :
> - fixing real sender with SRS [1].
SRS was NEVER approved to an RFC. Does Google actual handle it properly without
violating DMARC?

> - stop forwarding mail and do POP (gmail can do it) or IMAP from your
> favorite (web)mail client.
See prior in the thread, that this is NOT feasible for many users.

> Dmarc dns entry with report activated can help you understand why google
> blacklist you.
We are NOT blacklisted. We are throttled, and there is a major
difference there.

A62D234090F     4425 Mon May 11 17:19:24  bugzilla-daemon@gentoo.org
(host gmail-smtp-in.l.google.com[2607:f8b0:400e:c02::1a] said: 421-4.7.0
[2001:470:ea4a:1:214:c2ff:fe64:b2d3      15] Our system has detected 421-4.7.0
an unusual rate of unsolicited mail originating from your IP address. 421-4.7.0
To protect our users from spam, mail sent from your IP address has 421-4.7.0
been temporarily rate limited. Please visit 421-4.7.0
http://www.google.com/mail/help/bulk_mail.html to review our Bulk 421 4.7.0
Email Senders Guidelines. k5si11246054pdl.3 - gsmtp (in reply to end of DATA
command))
                                         ${CENSORED}@gmail.com

[1] I previously wrote about how this breaks lists:
http://robbat2.dreamwidth.org/238457.html

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Eray Aslan]

On Mon, May 11, 2015 at 04:47:31PM -0400, Michael Orlitzky wrote:
> On 05/11/2015 04:08 PM, Robin H. Johnson wrote:
> > By drop, I will clarify that they should ideally be rejected at SMTP
> > time, not silently dropped.
> 
> I believe those logs show a rejection after the message has been
> accepted initially (if I'm wrong, you can ignore the rest of this).

The analysis is correct.  Pre-queue filtering will help as we can safely
-meaning without causing backscatter- lower the threshold we reject spam
at.  There will still be some spam making its way to gmail but perhaps
it will be low enough to stay under gmail's radar.

The correct solution is to stop forwarding spam and the easiest way is
just stopping forwarding.  There are valid policy reasons for not going
that route but continuing forwarding because it is too difficult to
configure gmail is, well, not something I'd be comfortable with.  I do
expect more from gentoo devs.

In this case (in most cases?), infra should not be looking for consensus
but rather do what is right.

Anyway, I believe infra has all the info it needs at this point and I am
fine with whatever decision they make.

-- 
Eray

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach]

[-- Attachment #1: Type: text/plain, Size: 7567 bytes --]

Am Montag, 11. Mai 2015, 20:36:18 schrieb Robin H. Johnson:
> There are people that still accept mail that violates standards?
yes,
and there are mail sites and/or mail clients sending standard violating emails.

But the more truth is that there are many points within standards which are interpreted differently from different peoples / groups (or even mailer software developers) and there is no real clear / hard "border" what is a violence (and "could be dropped") and what is not - at least if you did not want to loose ham traffic for your users.

The email oecosystem does not dpend from a single RFC today - more and more basic parts of existing internet mail and it's features are defined in further RFCs or are conclusive from each other.

Two very typical examples:

1.) The sender domain has no MX nor abuse contact (i.e. RFC 2142)
Many pro level mass mailers do not have an "working" abuse contact, but there are still many smaller sites out which doesnt have too (because of limited DNS access or lack of knowledge). Dropping mail from such sites will lead you to loosing mails (even if it "just" hits one in thousand ham mails).

2.) BCC Header
Most Mailers today are filtering out BCC recipient headers at some point while this is not defined in the RFCs and still discussed hardly how far the deletion of BCC headers are breaking standards, resulting in possible lost of emails. See i.e. Phillip Hazels (EXIMs) statements in the net.


> My above statement is for mail that we ACCEPTED. If it violates
> standards, it's already denied at SMTP time.
hmmm,
you mean some more to very basic points of the standards.
 
> smtpd_restriction_classes = restrictive,permissive
> restrictive =
>     reject_invalid_hostname
>     reject_non_fqdn_hostname
>     reject_non_fqdn_recipient
>     reject_non_fqdn_sender
>     reject_unknown_sender_domain
>     reject_unknown_recipient_domain
>     check_sender_mx_access cidr:/etc/postfix/bogus_mx_records
>     check_sender_access pcre:/etc/postfix/sender_access_control.pcre
>     check_sender_access pcre:/etc/postfix/sender_access_control-aliases.pcre
> check_helo_access pcre:/etc/postfix/helo_checks
>     reject_unverified_sender
>     check_client_access cidr:/etc/postfix/filter.cidr
>     permit
> permissive =
>     permit


If it helps you a bit further, i can explain the basics of our setup, developed over nearly 20 years now, handling just a few hundredthousands smtp sessions per day and having NO spam folder or similiar (which would not save any time of the email user at the end) - but easily could be ad[a|o]pted per i.e. SIEVE to lead out less hard / more "unclear" spam into folders (i.e. instead of that mail where we make greylisting usually).

Because sendmail and postfix was to ressource inefficient for us sometimes in the early stages, we decided to go to EXIM (Phillip Hazel) - an own build optimized for our needs - including even some own mods today.

We avoided running SA from Amavis because of inefficiency.

Until today our incoming path goes:

 - EXIM with EXIM SA at SMTP time

Means we use Spamassassin directly at SMTP time, which allows us to dynamically "react" or further actively investigate a incoming smtp session if required. SA is only invoked for non authenticated mail over network btw..

Before exim contacts spamassassin at this stage, we run a bunch of checks in EXIM similiar to yours above, but some more (see down) which drop the connection or write data for further processing into the headers. If the connection is still alive, we run a hand full of RDNSBL checks, which "could reject" the session and then a hand full which just writes warnings into headers plus data for further processing steps.

If the sessions still "lives", EXIM contacts Spamassassin over socket and 

Here we have 3 "routes":

	- low spam -> Mails is going trough DIRECTLY
more then SA 2.3/3.0 - possible spam -> Greylisting (3 times TEMP Reject)
more then SA 5.2 - spam -> REJECT
more then SA 33.0 - blackhole

-> This kind of REJECT hits around 5 - 10% of spam connections, all other spam is usually catched before without the full email / mail body recieved.

Greylisting is "remembering" each contact<->contact handle and "quasi whitelists" the sender email after greylisting once to avoid further delays in the future - this helps very well for mail sites and/or clients which uses mail systems with bad reputation while "working OK".

SA EXIM is able to do teergrubing as well, but we did not use it in most situations - except partly in dictionary attacks.

At this point, parts of the mail traffic is going to an AMAVIS-NG for virus filtering only (user decide for it byself here) - no SA or RDNSBL again / at this place.

EXIM SA is no longer maintained officially, so we maintain it byself into actual EXIM source trees (would be nice to get it into Gentoos EXIM ebuild - i.e. by a USE flag - would help here if someone is interested - and if someone has a newer, at least same efficient solution it would be nioce to know).
 


Overview of EXIM checks of incoming SMTP sessions (parts of this are implemented in your postfix rules too):
--- snip ---

= HELO/EHLO required by SMTP RFC  See http://www.syndicat.com/faq/email/no_helo/
= Forged IP detected in HELO (it's mine) - $sender_helo_name  See http://www.syndicat.com/faq/email/forged_ip/
= Forged IP detected in HELO: $sender_helo_name
= Forged IP detected in HELO - $sender_helo_name != $sender_host_address  See http://www.syndicat.com/faq/email/forged_ip/
= Forged hostname detected in HELO - you are not $sender_helo_name See http://www.syndicat.com/faq/email/forged_ip/
= HELO is our IP
= $sender_helo_name is a silly HELO.
= RFC 1918 IP address in HELO ( See http://www.syndicat.com/faq/email/rfc1918-helo/ )
= $sender_address_domain is a silly domain. (i.e. localhost)
= HELO should be hostname but is $sender_helo_name . ( See http://www.syndicat.com/faq/email/helo_nohostname/ )
= HELO should be Fully Qualified Domain Name Host.Domain.Tld ( See RFC821 or http://www.syndicat.com/faq/email/helo_nofqdn/ )
= Forged hostname detected in HELO - $sender_helo_name is one of our domains
= Only one recipient accepted for NULL sender
= (DROP) too many unknown users (${eval:$rcpt_fail_count+1} failed recipients)
= Dictionary attack (${eval:$rcpt_fail_count+1} failed recipients).
=> Teergrube: dictionary attack (${eval:$rcpt_fail_count+1} failed recipients)
= unknown user
= X-Broken-Reverse-DNS: no DNS for IP address $sender_host_address
= acl_mail: (WARN-ONLY) Cannot reverse DNS $sender_host_address
= X-Broken-Reverse-DNS: no DNS for IP address $sender_host_address
= Content Policy Restriction: Mails to undisclosed recipients are not permitted.
= No contact MX - rfc-ignorant host $sender_host_name $sender_host_address . ( See http://www.syndicat.com/faq/email/rfc_ignorant/ )
= (WARN-ONLY, no reliable check possible) No MX abuse contact - rfc-ignorant host $sender_host_name $sender_host_address . ( See http://www.syndicat.com/faq/email/rfc_ignorant/ )
--- snap ---

then

RDNSBL:

deny = sbl-xbl.spamhaus.org : cbl.abuseat.org : zen.spamhaus.org : b.barracudacentral.org : psbl.surriel.com : ix.dnsbl.manitu.net
warn = dnsbl-3.uceprotect.net : ubl.unsubscore.com : dnsbl-1.uceprotect.net : dnsbl.sorbs.net


hth a bit.


cheerioh,


Niels.
-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 




[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: [gentoo-project] Anti-spam changes: proposal to drop spammy mail

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

hmm, of the ~90k messages i have in my local dir from ~7 years, it looks like 
~20 would have been incorrectly thrown away.  i can live with that.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

"Robin H. Johnson" <robbat2@gentoo.org> writes:
> TL;DR: As of May 17, @gentoo.org will drop incoming spammy mail
> instead of delivering it. Speak now or hold your peace.

Please no.  Even Gmail has sometimes false positives.  It is good enough
that Spamassassin marks mails and every user can set up rule either to
delete or to put to specific maildir.

Moreover one sometimes gets mail that sounds like advertisement but is
for example an amazing job offer.  I wouldn't like to miss that one.


-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Tue, May 12, 2015 at 1:19 AM, Eray Aslan <eras@gentoo.org> wrote:
>
> The correct solution is to stop forwarding spam and the easiest way is
> just stopping forwarding.  There are valid policy reasons for not going
> that route but continuing forwarding because it is too difficult to
> configure gmail is, well, not something I'd be comfortable with.  I do
> expect more from gentoo devs.

Configuring gmail to use POP isn't hard per-se, but it has a lot of limitations.

First, there is latency - they basically poll when they want to poll.
I find myself hitting refresh all the time as a result so that I don't
wait an hour to get my mail.

Another issue is that they won't use TLS/SSL unless they trust the
certificate, and there is no way to override this.  So, your options
are credentials possibly going out in plaintext, pay for a
certificate, or use a cert provider who won't revoke a certificate
even after pointing to private keys posted on github.  I suspect this
won't be a problem for retrieving mail from Gentoo, but it is one of
the reasons that I was desperately trying to forward mail to them.  In
the end I ended up switching to polling as a result of DKIM and
GMail's spam filters.

I find email an incredibly frustrating experience all-around.  It
works great as long as everybody doesn't use anybody for hosting who
isn't in the top-10 provider list, and doesn't use a mailing list.
Otherwise you get snared in the network of anti-spam tactics that only
spammers have the time to figure out how to avoid.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Peter Stuge]

Rich Freeman wrote:
> I find email an incredibly frustrating experience all-around.  It
> works great as long as everybody doesn't use anybody for hosting who
> isn't in the top-10 provider list, and doesn't use a mailing list.

DMARC marks top-10 essentially creating their own walled email garden.

It will only get worse.

One might even think that they don't like open source projects and
as much as they like their product (oops, I mean their users).


//Peter

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach]

[-- Attachment #1: Type: text/plain, Size: 2499 bytes --]

Am Dienstag, 12. Mai 2015, 06:26:41 schrieb Rich Freeman:
> I find email an incredibly frustrating experience all-around.  It
> works great as long as everybody doesn't use anybody for hosting who
> isn't in the top-10 provider list, and doesn't use a mailing list.
This is NOT true!

Our mail systems (we are under "top 10.000" to "100.000" i assume) have a top 
(and sometimes better) "reputation" and there is NO mail from our customers 
(except for real (!) abusive mail, if not blocked by us before) which is NOT 
reaching any working mail ISP on this globe and vice versa and our customers 
have a very good anti spam satisfaction as requested from us typically. The 
did not want to waste lifetime for handling spam or requesting recievers for 
"is my mail arrived yet" or similiar stuff - but do pay for such a service.

And there still ARE many others out who work at the same quality level, even 
on other customer fields, but there still ARE peoples offering mail service 
without knowing what they doing - independent from their cheap or expensive 
pricings. And there are some even "no-cost" providers out able to do that too 
(usually financing by service extensions or ads) - but usually larger then.

The ONLY requirement is a mail provider who does his job at an professional 
level . means: he has an admin who knows what he is doing and has the time to 
do that regularly. I remember many MS Exchange installations acting as MX / 
SMTP / MTA for a hand full of users - just setting up an Mailserver is easy, 
but running a reliable mail service is more then setup and let it run...

Email - as a major application "part" of the "internet" - is a complex 
ecosystem today and it takes ressources as such - like i.e. in full table BGP 
where it is usually a bad idea to run a full table BGP router with typical 
higher availability requirements byself without an competent person available 
24/7/365 for it (but there are manies doing this until today too and wonder if 
it results in bad customer satisfaction...).

But the time where "i set up a public internet mail server today" from zero is 
over - and i'm not very frustrated about this, because WE as the mail 
providers had to fiddle with the crap and misconfigs of others plus their 
resulting abusive traffic.

just my two cents,


Niels.

 
-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 




[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[J. Roeleveld]

On 11 May 2015 15:59:40 CEST, Rich Freeman <rich0@gentoo.org> wrote:
>On Mon, May 11, 2015 at 9:37 AM, C Bergström <cbergstrom@pathscale.com>
>wrote:
>> Sorry to shoot and run, but I think you're trying to tackle this
>> problem in the wrong way. The problem isn't to drop the mail. The
>> solution is to change email hosting providers. As a non-profit I
>> believe Google hosted apps would be an option (free).
>
>In general we try to stick to our social contract, and that means
>trying to avoid depending on proprietary technologies such as gmail.
>
>Now, I could see just using a FOSS-based IMAP/SMTP/POP provider,
>perhaps which allows things like forwarding and such, which allows us
>to have a copy of all our configuration and such in case we want to
>migrate.  I'm not super-familiar with the wordpress.com model but
>something like that also seems reasonable - we leverage donations of
>hosting services but we aren't bound to anything proprietary and have
>the ability to migrate off.
>
>I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
>is), and ditto for Google docs (or whatever the latest branding for
>that is). There is nothing magical about cloud-based services any more
>than there is anything magical about letting somebody else host your
>website.  The key is to ensure that the technologies are open so that
>you aren't bound to a single provider.

Rich,

If you are thinking of a FOSS email provider. Maybe investigate Fastmail?

They use postfix and cyrus. And they also handle a lot of the development of the latter.

Not sure if they would fit in with the rest, but I would trust them sooner then Google.

--
Joost 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[C Bergström]

On Sat, May 23, 2015 at 1:18 PM, J. Roeleveld <joost@antarean.org> wrote:
> On 11 May 2015 15:59:40 CEST, Rich Freeman <rich0@gentoo.org> wrote:
>>On Mon, May 11, 2015 at 9:37 AM, C Bergström <cbergstrom@pathscale.com>
>>wrote:
>>> Sorry to shoot and run, but I think you're trying to tackle this
>>> problem in the wrong way. The problem isn't to drop the mail. The
>>> solution is to change email hosting providers. As a non-profit I
>>> believe Google hosted apps would be an option (free).
>>
>>In general we try to stick to our social contract, and that means
>>trying to avoid depending on proprietary technologies such as gmail.
>>
>>Now, I could see just using a FOSS-based IMAP/SMTP/POP provider,
>>perhaps which allows things like forwarding and such, which allows us
>>to have a copy of all our configuration and such in case we want to
>>migrate.  I'm not super-familiar with the wordpress.com model but
>>something like that also seems reasonable - we leverage donations of
>>hosting services but we aren't bound to anything proprietary and have
>>the ability to migrate off.
>>
>>I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
>>is), and ditto for Google docs (or whatever the latest branding for
>>that is). There is nothing magical about cloud-based services any more
>>than there is anything magical about letting somebody else host your
>>website.  The key is to ensure that the technologies are open so that
>>you aren't bound to a single provider.
>
> Rich,
>
> If you are thinking of a FOSS email provider. Maybe investigate Fastmail?
>
> They use postfix and cyrus. And they also handle a lot of the development of the latter.
>
> Not sure if they would fit in with the rest, but I would trust them sooner then Google.

Trust? LOL - If by trust you mean the government can man-in-the-middle
attack them easily - then sure.. gmail always uses encryption.. does
fastmail force that as well? Google has a much stronger means to push
back short term and long term against government spying (snowden).
fastmail would have to comply just the same and if you go back in
history - you'll see other providers who didn' t comply and the only
outcome was for them to go out of business... (happened once?)

social contract shouldn't be a religious contract - It's not like I
ever suggested we use something which isn't blessed by the pope. I
guess I should just be quiet since everyone has their own religion...

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach (Syndicat.com)]

[-- Attachment #1: Type: text/plain, Size: 2423 bytes --]


> Am 23.05.2015 um 08:18 schrieb J. Roeleveld <joost@antarean.org>:
> 
>> I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
>> is), and ditto for Google docs (or whatever the latest branding for
>> that is). There is nothing magical about cloud-based services any more
>> than there is anything magical about letting somebody else host your
>> website.  The key is to ensure that the technologies are open so that
>> you aren't bound to a single provider.
> 
> Rich,
> 
> If you are thinking of a FOSS email provider. Maybe investigate Fastmail?

I think herein is a major misunderstanding.

There ARE excellent Email Providers for nearly any kind of customer and customer profile - and many of them are only working with open source tools (GPL and/or BSD licensed f.i.).

But if you are looking for a „cost free“ mail provider someone other has to pay for the service infrastructure and the skilled work and time experienced admins has to take in every day. At least Hardware and energy costs money on world market pricing, plus "cabling" etc. - even for Google and Co..

Gmail does this (like many others) by placing ads and using your mail content and social / communication relationships to sell „statistics“ and more to third parties. If this fits YOU expectations, THIS is the correct product for yours.

If you are willing to pay for a high quality email service you will find many really good products on the market, without ads and with respect for your possibly privacy needs (if you need such...) - fitting your needs depending from your expectations in detail.

Most (private) peoples want „cost free“ email services - leading to that market of free and widely less quality mail services or give that job to peoples (i.e. a web agency which is "hosting provider btw“) who don’t have the abilities to run Email on such a professional level.

btw: We switched away from sendmail and postfix later around 98 because of their inefficient ressource footprint for large scale email setups. Some of the very large mailers today still holding on postfix usually have developed their own strongly modified and mostly („crapped") "version" of it (more or less giggling around GPL barriers here) - this is not THAT postfix anymore most people get with their linux distri. ß)


just my two cents,

cheerioh,



Niels.
—
http://www.syndicat.com

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 831 bytes --]

[gentoo-dev] Re: Anti-spam changes: proposal to drop spammy mail

[Duncan]

Niels Dettenbach (Syndicat.com) posted on Sat, 23 May 2015 08:39:36 +0200
as excerpted:

>  Some
> of the very large mailers today still holding on postfix usually have
> developed their own strongly modified and mostly („crapped") "version"
> of it (more or less giggling around GPL barriers here) - this is not
> THAT postfix anymore most people get with their linux distri. ß)

FWIW on those GPL barriers... and with the usual "no lawyer here" 
disclaimer...

It's worth noting the difference between the GPL and the AGPL, the latter 
of which considers usage of a server-based service to be distribution of 
that service, thereby triggering the traditional GPL sources distribution 
requirements.

The GPL, by contrast, normally applies only locally, making the company 
doing the mods the only direct user, and they normally have access to the 
sources already, since they're either making the mods or commissioning 
them, themselves.

Thus it is that many cloud-based services can legally avoid the otherwise 
restrictions of the GPL, because they are their own user and are not 
considered to be distributing.  But the AGPL, unlike the GPL, never 
really developed a following of critical mass to create a self-sustaining 
ecosystem.

So it isn't that these providers are giggling around the GPL 
restrictions.  Those restrictions simply don't apply to the license 
chosen for the apps and libs they tend to use, and if they did, because 
the AGPL never developed that strong an ecosystem, it's always reasonably 
simple to simply go with a different "open source", or even "free 
software" alternative.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Sat, 23 May 2015 08:18:28 +0200
"J. Roeleveld" <joost@antarean.org> wrote:

> If you are thinking of a FOSS email provider. Maybe investigate
> Fastmail?
> 
> They use postfix and cyrus. And they also handle a lot of the
> development of the latter.
> 
> Not sure if they would fit in with the rest, but I would trust them
> sooner then Google.

I refuse to use Gmail but I have trusted FastMail for about 13 years,
though I do pay them for the privilege. As they say, if you aren't
paying for the product then you are the product. Despite my preference
for FastMail, I don't forward my Gentoo mail and connect using IMAP
directly. I am totally fine with that setup. The spam is a tad annoying
but it's not reached intolerable levels for me and I do use procmail,
which helps a little.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

On Sat, 23 May 2015 13:24:11 +0700 C Bergström wrote:
> On Sat, May 23, 2015 at 1:18 PM, J. Roeleveld <joost@antarean.org> wrote:
> > Rich,
> >
> > If you are thinking of a FOSS email provider. Maybe investigate Fastmail?
> >
> > They use postfix and cyrus. And they also handle a lot of the development of the latter.
> >
> > Not sure if they would fit in with the rest, but I would trust them sooner then Google.
> 
> Trust? LOL - If by trust you mean the government can man-in-the-middle
> attack them easily - then sure.. gmail always uses encryption.. does
> fastmail force that as well? Google has a much stronger means to push
> back short term and long term against government spying (snowden).

Oh, really? If you have read materials showed by Snowden, then you
should know that Google is already hooked to its deps by (at least)
all interested us agencies. They have internal encryption? So what?
They just gave up keys or provided direct access to depcryption
stations or by whatever other means granted access to demanding us
agencies.

> fastmail would have to comply just the same and if you go back in
> history - you'll see other providers who didn' t comply and the only
> outcome was for them to go out of business... (happened once?)
> 
> social contract shouldn't be a religious contract - It's not like I
> ever suggested we use something which isn't blessed by the pope. I
> guess I should just be quiet since everyone has their own religion...

Its not a religion, its a protection of project's freedom. If you
don't care about freedom, that's your own personal right. But this
doesn't mean that other people must gave up their freedom for the
same of convenience of others.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Sat, May 23, 2015 at 2:18 AM, J. Roeleveld <joost@antarean.org> wrote:
> On 11 May 2015 15:59:40 CEST, Rich Freeman <rich0@gentoo.org> wrote:
>>
>>I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
>>is), and ditto for Google docs (or whatever the latest branding for
>>that is). There is nothing magical about cloud-based services any more
>>than there is anything magical about letting somebody else host your
>>website.  The key is to ensure that the technologies are open so that
>>you aren't bound to a single provider.
>
> Rich,
>
> If you are thinking of a FOSS email provider. Maybe investigate Fastmail?
>
> They use postfix and cyrus. And they also handle a lot of the development of the latter.
>

My mail all goes through my own postfix server and POP/IMAP server
before it gets to Gmail, and I already have an alternative solution
for outbound SMTP for this server.

I was talking about a decent FOSS browser-based MUA.  The only ones
I'm aware of are Roundcube and Squirrelmail, and neither supports
keyboard shortcuts or tag-based mail as far as I'm aware.  Actually,
I'm not aware of any FOSS IMAP implementation that supports tagging -
that is an email can be in more than one "folder" at the same time.
But, I haven't looked too closely into that since without an MUA it
isn't terribly useful.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Sat, 23 May 2015 07:16:10 -0400 Rich Freeman wrote:
> On Sat, May 23, 2015 at 2:18 AM, J. Roeleveld <joost@antarean.org> wrote:
> > On 11 May 2015 15:59:40 CEST, Rich Freeman <rich0@gentoo.org> wrote:
> >>
> >>I'd REALLY like to see a FOSS alternative to Gmail (a good one, that
> >>is), and ditto for Google docs (or whatever the latest branding for
> >>that is). There is nothing magical about cloud-based services any more
> >>than there is anything magical about letting somebody else host your
> >>website.  The key is to ensure that the technologies are open so that
> >>you aren't bound to a single provider.
> >
> > Rich,
> >
> > If you are thinking of a FOSS email provider. Maybe investigate Fastmail?
> >
> > They use postfix and cyrus. And they also handle a lot of the development of the latter.
> >
> 
> My mail all goes through my own postfix server and POP/IMAP server
> before it gets to Gmail, and I already have an alternative solution
> for outbound SMTP for this server.
> 
> I was talking about a decent FOSS browser-based MUA.  The only ones
> I'm aware of are Roundcube and Squirrelmail, and neither supports
> keyboard shortcuts or tag-based mail as far as I'm aware.  Actually,
> I'm not aware of any FOSS IMAP implementation that supports tagging -
> that is an email can be in more than one "folder" at the same time.
> But, I haven't looked too closely into that since without an MUA it
> isn't terribly useful.

Sylpheed supports filters which allow you to have e-mails in
multiple directories based on arbitrary user-defined filtering.
It supports IMAP also, though I never use it as I prefer POP3 and
SMTP.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Sat, May 23, 2015 at 8:32 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>
> Sylpheed supports filters which allow you to have e-mails in
> multiple directories based on arbitrary user-defined filtering.
> It supports IMAP also, though I never use it as I prefer POP3 and
> SMTP.
>

Well, besides not being browser-based as far as I can tell, without
integration with the IMAP server those emails in multiple directories
won't show up in multiple directories when accessed from any other
client.  What I like about Gmail is that I can operate from the
browser, but still have access to my mail via IMAP if I need it, and
of course it has a really nice Android offline client (and an offline
html5 client as well).  These days IMAP clients on Android are
probably a lot better, so the android client isn't as important as it
used to be most likely, but I'd still like contacts to be in-sync
across all browser and android clients (which isn't something IMAP
alone can deliver).

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach (Syndicat.com)]

[-- Attachment #1: Type: text/plain, Size: 3476 bytes --]


> Am 23.05.2015 um 15:07 schrieb Rich Freeman <rich0@gentoo.org>:
> 
> Well, besides not being browser-based as far as I can tell, without
> integration with the IMAP server those emails in multiple directories
> won't show up in multiple directories when accessed from any other
> client.
This is a behavior of your email client and typically not part of the service (except if you see webmail clients as part of it).

>  What I like about Gmail is that I can operate from the
> browser, but still have access to my mail via IMAP if I need it, and
> of course it has a really nice Android offline client (and an offline
> html5 client as well).
WOW,
this is what our customers still had around 20 years ago and far before any "google" exist...

>  These days IMAP clients on Android are
> probably a lot better, so the android client isn't as important as it
> used to be most likely, but I'd still like contacts to be in-sync
> across all browser and android clients (which isn't something IMAP
> alone can deliver).

If your mail provider offers you "real“ IMAP and protocol standards around (i.e. SIEVE etc.) (and not that crap most freeman providers offer - including gmail) this is fully the question of YOUR email client.

Most professional mail providers has to offer at least one (usually more then one) webmail GUI / interface for their users / customers and are open and fast enough to even able to serve any otherwhere hosted third party „webmail" client (if someone really need that because of very specialized needs / expectations to his „webmail“). Many of that solutions allow - if users really need / want this beside any data protection laws and structures (you have to hand over a personal (!!!) password to a third party - and this beside the fact that that party is allowed from you to use that for nearly „anything“...) - to view and work with different IMAP accounts on different locations / providers as well.

This means users can decide between i.e. a „simple“ platform independent webmail only GUI over PIM solutions up to full scale Groupware / PIM solutions including group management and higher level administration levels. Without or with further synchronization and/or „cloud“ services. Users can generate multiple mail addresses as needed, have not a hardly downcropped SMTP / MTA service which allows just some MB per Email or several hundred emails by day or similar limits. Nearly the whole „cloud storage“ Dropbox mania was a result of kiddingly email services allowing mails up to 5 - 50 MB size or so - limits which are not acceptable for any more professional user (but even crazy for end users who want to send some files to their recipients in a mail).

My experience is that more then 99% of all email users did not know/recognized the power of real IMAP - resultet from crippled IMAP services of their providers or still being in POP3 in their mind. Services like server side searching, access rules to folders or even the SIEVE filter standards are very new to them, but available since around 20 years on the email service market. Gmail was one of the first „fee free“ mail services internationally offering at least a halfway usable IMAP - 20 years after that was standard for pro users and many of the features that customer audience is calling „gmail feature“ are still parts of standards, available decades before...

this just btw…


cheerioh,


Niels.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 831 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Sat, May 23, 2015 at 9:34 AM, Niels Dettenbach (Syndicat.com)
<nd@syndicat.com> wrote:
>
>> Am 23.05.2015 um 15:07 schrieb Rich Freeman <rich0@gentoo.org>:
>>
>> Well, besides not being browser-based as far as I can tell, without
>> integration with the IMAP server those emails in multiple directories
>> won't show up in multiple directories when accessed from any other
>> client.
> This is a behavior of your email client and typically not part of the service (except if you see webmail clients as part of it).

I almost exclusively access my email via the web, but also desire to
access it via clients, such as on mobile.

With Gmail I can have an email with 14 tags.  Via IMAP it shows up as
if it were in 14 folders at the same time.  It is a bit kludgy, but it
at least works.

If I just used squirrelmail and sylpheed as clients, and courier as my
IMAP server, then as far as I can tell if I did tag an email with 14
tags in sylpheed then I wouldn't see that in squirrelmail.

>
>>  What I like about Gmail is that I can operate from the
>> browser, but still have access to my mail via IMAP if I need it, and
>> of course it has a really nice Android offline client (and an offline
>> html5 client as well).
> WOW,
> this is what our customers still had around 20 years ago and far before any "google" exist...

Not with tagging, as far as I'm aware.  I also doubt that you had an
offline html5 client 20 years ago.

>

> My experience is that more then 99% of all email users did not
> know/recognized the power of real IMAP - resultet from crippled
> IMAP services of their providers or still being in POP3 in their
> mind. Services like server side searching, access rules to folders or
> even the SIEVE filter standards are very new to them, but available
> since around 20 years on the email service market. Gmail was one of
> the first „fee free“ mail services internationally offering at
> least a halfway usable IMAP - 20 years after that was standard for
> pro users and many of the features that customer audience is calling
> „gmail feature“ are still parts of standards, available decades
> before...

Sure, and that was how I was doing it before I switched to Gmail.
Before I had a smartphone/chromebook/etc I'd just check email from
thunderbird, and have squirrelmail available for rare web-only use.
Then my usage patterns changed and I don't even have an X11-based
email client anymore.  All those rich client features don't work when
they're chained to your home directory and don't sync their settings
with all the other clients you use.

Gmail draws the boundaries different between client/server on mail,
and I find it works much better for me that way.

And to be useful I also need reasonable integration across
mail/contacts/calendar/docs.  Really to switch back to FOSS I need
something with near-feature-parity with Google-everything, because
that is already fairly minimalist.

There are plenty of things that annoy me with Gmail, and I'd be glad
to ditch it in a heartbeat.  I've just not found anything I can host
myself which is comparable.

-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Sat, 23 May 2015 15:34:28 +0200
"Niels Dettenbach (Syndicat.com)" <nd@syndicat.com> wrote:
> >  What I like about Gmail is that I can operate from the
> > browser, but still have access to my mail via IMAP if I need it, and
> > of course it has a really nice Android offline client (and an
> > offline html5 client as well).
> WOW,
> this is what our customers still had around 20 years ago and far
> before any "google" exist...

But Google doesn't make me change my email address every two years as I
move, or as ISPs get bought out or go bust.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach (Syndicat.com)]

[-- Attachment #1.1: Type: text/plain, Size: 652 bytes --]


> Am 23.05.2015 um 16:23 schrieb Ciaran McCreesh <ciaran.mccreesh@googlemail.com>:
> 
> But Google doesn't make me change my email address every two years as I
> move, or as ISPs get bought out or go bust.

Think a bit about your own non-sense:

1.) gmail is much younger then many of really professional mail providers today.

2.) if you register your OWN domain, you can use it with ANY professional email provider and it leave YOUrs your life long independent from any mail provider - move it wherever you need anytime… It is NEVER a good idea to think a lended email from anyone as a „life long“...


cheerioh,


Niels.



[-- Attachment #1.2: Type: text/html, Size: 2478 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 831 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach (Syndicat.com)]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]


> Am 23.05.2015 um 16:20 schrieb Rich Freeman <rich0@gentoo.org>:
> 
> With Gmail I can have an email with 14 tags.  Via IMAP it shows up as
> if it were in 14 folders at the same time.  It is a bit kludgy, but it
> at least works.
This is NOT part of a mail service - it is part of a mail client. There are different webmail solutions out (usable with any real mail provider) allowing unlimited tagging btw...

It is typically not the wisest idea to choose a mail service by any kind of client software it offers - this remembers me hardly to the Lotus notes hype…


cheerioh,


Niels.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 831 bytes --]

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Rich Freeman]

On Sat, May 23, 2015 at 10:32 AM, Niels Dettenbach (Syndicat.com)
<nd@syndicat.com> wrote:
>
>> Am 23.05.2015 um 16:20 schrieb Rich Freeman <rich0@gentoo.org>:
>>
>> With Gmail I can have an email with 14 tags.  Via IMAP it shows up as
>> if it were in 14 folders at the same time.  It is a bit kludgy, but it
>> at least works.
> This is NOT part of a mail service - it is part of a mail client. There are different webmail solutions out (usable with any real mail provider) allowing unlimited tagging btw...
>

Like I said, Gmail draws the line between client and server
differently, and I find it more useful.

Are any of those webmail solutions FOSS, and do they support accessing
the mail by tags via IMAP using arbitrary clients?  Do they support
tagging via keyboard shortcuts alone, at least as far as applying the
trash tag, removing the inbox tag, and applying the spam tag go?  Do
they also integrate with calendar/contact/document servers?  Heck, I'm
not even aware of a decent open calendar service protocol.  Sure,
everybody and their uncle can export ics files, but that is one-way
only.

I find those Gmail features useful.  You can argue that exposing tags
via IMAP violates the original intent of IMAP, but that doesn't make
it less useful.  It is like arguing that ZFS is a rampant layering
violation - that is a nice philisophical argument but not nearly as
nice as avoiding the write hole or the need to do a read before
writing a partial stripe.

> It is typically not the wisest idea to choose a mail service by any kind of client software it offers - this remembers me hardly to the Lotus notes hype…

Sure, and if anybody made a decent webmail client with the features
above that wasn't bundled to a service I'd happily use it.


-- 
Rich

Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

On 23 May 2015 16:29, Niels Dettenbach (Syndicat.com) wrote:

i hope we also update the server to reject e-mails to mailing lists that include 
html nonsense
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]