[gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[Paul Hartman]

Hi,

The background:

Long-time Gentoo user, newbie to mail server setup. I've got a Gentoo
server (~amd64) and I set up Postfix and Dovecot following this guide:
http://en.gentoo-wiki.com/wiki/Mail_server_using_Postfix_and_Dovecot

I'm using postfix-2.8.4 and dovecot-2.0.13

So far, everything works as advertised. I've added a domain, created
accounts, additionally I've set up sqlgrey, DKIM signing and SPF,
sender-ID and all of that good stuff. My sent mail goes out
successfully and incoming mail gets delivered to the virtual mailboxes
and everything's working properly as far as I can tell.

The problem:

My existing (hosted) mail server has a catch-all account so any mail
to the domains I own goes to me. I believe they use qmail. The
delivered-to header on the messages retain the address of the original
message, not the catch-all address to which it is forwarded. Why do I
care? I basically use a different address for every site or person I
interact with and then sort them out into folders based on the
delivered-to address. (For example if I bought something from Sears I
would use sears@example.com as my email address).

With postfix, it doesn't work that way. I set up a catchall, but the
delivered-to: header becomes catchall@example.com for all of the
catchall mail. I can't change my rules to use use the To: address
because it is not reliable (BCC, spam...).

The idea:

Assuming I can't do anything about how Postfix handles the
Delivered-To header, I'd like to insert a new header entry
(X-Originally-To: or something like that) into incoming mail before it
hits the catchall forward, so I can know to whom the email was
originally addressed... but i don't really know where to begin.

The question:

Are there any postfix gurus out there who can point me in the right
direction? Thanks in advance for any tips or advice (or if you want to
tell me that I'm doing it all wrong).

Thanks,
Paul

Re: [gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[kashani]

On 7/7/2011 2:50 PM, Paul Hartman wrote:
> The idea:
>
> Assuming I can't do anything about how Postfix handles the
> Delivered-To header, I'd like to insert a new header entry
> (X-Originally-To: or something like that) into incoming mail before it
> hits the catchall forward, so I can know to whom the email was
> originally addressed... but i don't really know where to begin.
>
> The question:
>
> Are there any postfix gurus out there who can point me in the right
> direction? Thanks in advance for any tips or advice (or if you want to
> tell me that I'm doing it all wrong).

It should already be there at least in 2.7.4 which is stable unless 
you've really tweaked your main.cf. I'd run a postconf | grep 
enable_orig and see if it's not set to yes.

kashani

http://www.postfix.org/postconf.5.html

enable_original_recipient (default: yes)

     Enable support for the X-Original-To message header. This header is 
needed for multi-recipient mailboxes.

     When this parameter is set to yes, the cleanup(8) daemon performs 
duplicate elimination on distinct pairs of (original recipient, 
rewritten recipient), and generates non-empty original recipient queue 
file records.

     When this parameter is set to no, the cleanup(8) daemon performs 
duplicate elimination on the rewritten recipient address only, and 
generates empty original recipient queue file records.

     This feature is available in Postfix 2.1 and later. With Postfix 
version 2.0, support for the X-Original-To message header is always 
turned on. Postfix versions before 2.0 have no support for the 
X-Original-To message header.

Re: [gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[Paul Hartman]

On Thu, Jul 7, 2011 at 5:25 PM, kashani <kashani-list@badapple.net> wrote:
> On 7/7/2011 2:50 PM, Paul Hartman wrote:
>>
>> The idea:
>>
>> Assuming I can't do anything about how Postfix handles the
>> Delivered-To header, I'd like to insert a new header entry
>> (X-Originally-To: or something like that) into incoming mail before it
>> hits the catchall forward, so I can know to whom the email was
>> originally addressed... but i don't really know where to begin.
>>
>> The question:
>>
>> Are there any postfix gurus out there who can point me in the right
>> direction? Thanks in advance for any tips or advice (or if you want to
>> tell me that I'm doing it all wrong).
>
> It should already be there at least in 2.7.4 which is stable unless you've
> really tweaked your main.cf. I'd run a postconf | grep enable_orig and see
> if it's not set to yes.
>
> kashani
>
> http://www.postfix.org/postconf.5.html
>
> enable_original_recipient (default: yes)
>
>    Enable support for the X-Original-To message header. This header is
> needed for multi-recipient mailboxes.

Hi kashani,

I actually read about that option when I was trying to make this
happen (forgot about it when composing my original message). Googling
that option I found that most people were interested in combining
multi-recipient messages to one on disk (to save space). Indeed the
option is already set to "yes" on my setup, but I still don't get that
header. I supposed that it has nothing to do with the address I'm
interested in (from the envelope) and instead is looking at the To:
name from headers (which is unchanged). Or because my message does not
have multiple recipients. But maybe I'm completely misunderstanding
what it's all about.

Here's what an incoming mail to catchall looks like in my syslog:

Jul  4 22:11:10 virtual postfix/smtpd[3032]: connect from
mail-iw0-f174.google.com[209.85.214.174]
Jul  4 22:11:10 virtual sqlgrey: grey: domain awl match: updating
209.85.214(209.85.214.174), gmail.com
Jul  4 22:11:10 virtual postfix/smtpd[3032]: E66CA19F0D:
client=mail-iw0-f174.google.com[209.85.214.174]
Jul  4 22:11:10 virtual postfix/cleanup[3043]: E66CA19F0D:
message-id=<4E127EDC.6050308@gmail.com>
Jul  4 22:11:11 virtual postfix/qmgr[2633]: E66CA19F0D:
from=<paul.hartman@gmail.com>, size=1689, nrcpt=1 (queue active)
Jul  4 22:11:11 virtual dovecot: auth:
mysql(/var/run/mysqld/mysqld.sock): Connected to database postfixadmin
Jul  4 22:11:11 virtual dovecot: lda(catchall@example.com):
msgid=<4E127EDC.6050308@gmail.com>: saved mail to INBOX
Jul  4 22:11:11 virtual postfix/qmgr[2633]: E66CA19F0D: removed
Jul  4 22:11:11 virtual postfix/pipe[3045]: E66CA19F0D:
to=<catchall@example.com>, orig_to=<xyz@example.com>, relay=dovecot,
delay=0.22, delays=0.18
/0.01/0/0.03, dsn=2.0.0, status=sent (delivered via dovecot service)
Jul  4 22:11:41 virtual postfix/smtpd[3032]: disconnect from
mail-iw0-f174.google.com[209.85.214.174]

So it does run cleanup (and before passing the msg to dovecot) and
there is reference to orig_to so at least I know postfix realizes
there's a difference. Maybe this should be done by dovecot instead?

Here is my main.cf (comment blocks removed):

[cut]
queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib64/postfix
data_directory = /var/lib/postfix
mail_owner = postfix
myhostname = virtual.example.com
unknown_local_recipient_reject_code = 550
debugger_command =
         PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
         ddd $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/sbin/sendmail
newaliases_path = /usr/bin/newaliases
mailq_path = /usr/bin/mailq
setgid_group = postdrop
html_directory = /usr/share/doc/postfix-2.8.3-r2/html
manpage_directory = /usr/share/man
sample_directory = /etc/postfix
readme_directory = /usr/share/doc/postfix-2.8.3-r2/readme
home_mailbox = .maildir/

dovecot_destination_recipient_limit = 1
virtual_transport = dovecot

smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth

virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domain_maps.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf

smtpd_sender_restrictions=
 reject_non_fqdn_sender

smtpd_reject_unlisted_sender = yes

smtpd_recipient_restrictions=
 permit_mynetworks,
 permit_sasl_authenticated,
 reject_non_fqdn_recipient,
 reject_unauth_destination,
 check_policy_service inet:127.0.0.1:2501

smtpd_milters     = unix:/var/run/dk-filter/dk-filter.sock,
unix:/var/run/dkim-filter/dkim-filter.sock, inet:localhost:8026
non_smtpd_milters = unix:/var/run/dk-filter/dk-filter.sock,
unix:/var/run/dkim-filter/dkim-filter.sock, inet:localhost:8026
inet_protocols = all
[cut]

As I basically followed the wiki, I have no idea if this config is
ordinary or not. :) Thanks again for your help.

Thanks,
Paul

Re: [gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[kashani]

On 7/7/2011 4:59 PM, Paul Hartman wrote:
> On Thu, Jul 7, 2011 at 5:25 PM, kashani<kashani-list@badapple.net>  wrote:
>> On 7/7/2011 2:50 PM, Paul Hartman wrote:
>>>
>>> The idea:
>>>
>>> Assuming I can't do anything about how Postfix handles the
>>> Delivered-To header, I'd like to insert a new header entry
>>> (X-Originally-To: or something like that) into incoming mail before it
>>> hits the catchall forward, so I can know to whom the email was
>>> originally addressed... but i don't really know where to begin.
>>>
>>> The question:
>>>
>>> Are there any postfix gurus out there who can point me in the right
>>> direction? Thanks in advance for any tips or advice (or if you want to
>>> tell me that I'm doing it all wrong).
>>
>> It should already be there at least in 2.7.4 which is stable unless you've
>> really tweaked your main.cf. I'd run a postconf | grep enable_orig and see
>> if it's not set to yes.
>>
>> kashani
>>
>> http://www.postfix.org/postconf.5.html
>>
>> enable_original_recipient (default: yes)
>>
>>     Enable support for the X-Original-To message header. This header is
>> needed for multi-recipient mailboxes.
>
> Hi kashani,
>
> I actually read about that option when I was trying to make this
> happen (forgot about it when composing my original message). Googling
> that option I found that most people were interested in combining
> multi-recipient messages to one on disk (to save space). Indeed the
> option is already set to "yes" on my setup, but I still don't get that
> header. I supposed that it has nothing to do with the address I'm
> interested in (from the envelope) and instead is looking at the To:
> name from headers (which is unchanged). Or because my message does not
> have multiple recipients. But maybe I'm completely misunderstanding
> what it's all about.

I think I've got it figured out and this is your culprit.

http://en.gentoo-wiki.com/wiki/Mail_server_using_Postfix_and_Dovecot#Dovecot_Integration_-_LDA

This is because Postfix adds the x-original-to when it delivers, but not 
when it passes the mail via lmtp to Dovecot. See this Dovecot thread for 
some details.

http://www.dovecot.org/list/dovecot/2011-January/056787.html

The primary benefits of Dovecot LDA seem to be cache files and Sieve.
http://wiki.dovecot.org/LDA/Indexing
http://wiki.dovecot.org/LDA/Sieve

If you aren't using Sieve, I would try is using virtual_transport = 
virtual instead of virtual_transport = dovecot

You might need to change some settings in your Dovecot config to match 
where Postfix will deliver the emails, specifically mail_location = 
maildir:/var/mail/%d/%n/Maildir/:INDEX=/var/mail/%d/%n/indexes

This are the settings I use for Postfix w/ Courier. Should work with 
Dovecot, but again you might need to change things a bit.

# virtual stuff
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_gid_maps = static:207
virtual_mailbox_base = /var/mail/
virtual_mailbox_domains = 
proxy:mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_limit = 512400000
virtual_mailbox_maps = 
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_minimum_uid = 207
virtual_transport = virtual
virtual_uid_maps = static:207

kashani

Re: [gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[Paul Hartman]

On Thu, Jul 7, 2011 at 7:41 PM, kashani <kashani-list@badapple.net> wrote:
> On 7/7/2011 4:59 PM, Paul Hartman wrote:
>>
>> On Thu, Jul 7, 2011 at 5:25 PM, kashani<kashani-list@badapple.net>  wrote:
>>>
>>> On 7/7/2011 2:50 PM, Paul Hartman wrote:
>>>>
>>>> The idea:
>>>>
>>>> Assuming I can't do anything about how Postfix handles the
>>>> Delivered-To header, I'd like to insert a new header entry
>>>> (X-Originally-To: or something like that) into incoming mail before it
>>>> hits the catchall forward, so I can know to whom the email was
>>>> originally addressed... but i don't really know where to begin.
>>>>
>>>> The question:
>>>>
>>>> Are there any postfix gurus out there who can point me in the right
>>>> direction? Thanks in advance for any tips or advice (or if you want to
>>>> tell me that I'm doing it all wrong).
>>>
>>> It should already be there at least in 2.7.4 which is stable unless
>>> you've
>>> really tweaked your main.cf. I'd run a postconf | grep enable_orig and
>>> see
>>> if it's not set to yes.
>>>
>>> kashani
>>>
>>> http://www.postfix.org/postconf.5.html
>>>
>>> enable_original_recipient (default: yes)
>>>
>>>    Enable support for the X-Original-To message header. This header is
>>> needed for multi-recipient mailboxes.
>>
>> Hi kashani,
>>
>> I actually read about that option when I was trying to make this
>> happen (forgot about it when composing my original message). Googling
>> that option I found that most people were interested in combining
>> multi-recipient messages to one on disk (to save space). Indeed the
>> option is already set to "yes" on my setup, but I still don't get that
>> header. I supposed that it has nothing to do with the address I'm
>> interested in (from the envelope) and instead is looking at the To:
>> name from headers (which is unchanged). Or because my message does not
>> have multiple recipients. But maybe I'm completely misunderstanding
>> what it's all about.
>
> I think I've got it figured out and this is your culprit.
>
> http://en.gentoo-wiki.com/wiki/Mail_server_using_Postfix_and_Dovecot#Dovecot_Integration_-_LDA
>
> This is because Postfix adds the x-original-to when it delivers, but not
> when it passes the mail via lmtp to Dovecot. See this Dovecot thread for
> some details.
>
> http://www.dovecot.org/list/dovecot/2011-January/056787.html
>
> The primary benefits of Dovecot LDA seem to be cache files and Sieve.
> http://wiki.dovecot.org/LDA/Indexing
> http://wiki.dovecot.org/LDA/Sieve
>
> If you aren't using Sieve, I would try is using virtual_transport = virtual
> instead of virtual_transport = dovecot
>
> You might need to change some settings in your Dovecot config to match where
> Postfix will deliver the emails, specifically mail_location =
> maildir:/var/mail/%d/%n/Maildir/:INDEX=/var/mail/%d/%n/indexes
>
> This are the settings I use for Postfix w/ Courier. Should work with
> Dovecot, but again you might need to change things a bit.
>
> # virtual stuff
> virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
> virtual_gid_maps = static:207
> virtual_mailbox_base = /var/mail/
> virtual_mailbox_domains =
> proxy:mysql:/etc/postfix/mysql_virtual_domains_maps.cf
> virtual_mailbox_limit = 512400000
> virtual_mailbox_maps =
> proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
> virtual_minimum_uid = 207
> virtual_transport = virtual
> virtual_uid_maps = static:207

Really great info, thanks a lot for taking your time to look into it.
I will experiment with it and see what I can do. Thankfully I'm just
using a new domain for testing for now, so I have the freedom to break
things as much as needed until I get it right. :)

Thanks again,
Paul

Re: [gentoo-server] Postfix: Can I insert a custom header to incoming mail?

[Paul Hartman]

On Thu, Jul 7, 2011 at 7:41 PM, kashani <kashani-list@badapple.net> wrote:
> If you aren't using Sieve, I would try is using virtual_transport = virtual
> instead of virtual_transport = dovecot

Thanks! That seems to have worked. Mail is still being delivered, and
now the X-Original-To: header is in-place. Really great, now I can
start migrating my mail domains that depend on that feature.

I also found a semi-work-around posted by Wietse Venema on the postfix list:

[quote]
You can fake it in the SMTP server with

/etc/postfix/main.cf:
smtpd_recipient_restrictions =
check_recipient_access pcre:/etc/postfix/recipient_access.pcre

/etc/postfix/recipient_access.pcre
/(.+)/ prepend X-Original-To: $1
[end quote]

This method worked with virtual_transport = dovecot, however it had
problems dealing with multi-recipient mail. (for example if
user1@mydomain.com and user2@mydomain.com were both recipients on the
same message, both users' mail would have the first username as the
X-Original-To header). That situation is rare enough in my case that
it would have been acceptable if the virtual_transport = virtual
wasn't usable.

Thanks again,
Paul