For most gentoo users, a simple mail client and fetchmail will do. However, if you're hosting a domain with your system, you'll need a full blown MTA (Mail Transfer Agent). And if you're hosting multiple domains, then you'll definitely need something more robust to handle all of the email for your users. This system was designed to be an elegant solution to that problem.
A virtual mail system needs to be able to handle email for numerous domains with multiple users over a variety of interfaces. This presents some issues that must be dealt with. For instance, what if you have two users on different domains that want the same user name? If you are providing imap access and smtp-auth, how do combine the various authentication daemons into a single system? How do you provide security for the numerous components that comprise the system? How do you manage it all?
This howto will show you how to set up with a mail system capable of handling mail for as many domains as your hardware can handle, supports virtual mail users that don't require shell accounts, has domain specific user names, can authenticate web, imap, smtp, and pop3 clients against a single database, utilizes ssl for transport layer security, has a web interface, can handle mailing lists for any domain on the machine, and is controlled by a nice, central and easy mysql database.
There are quite a variety of ways to go about setting up a virtual mailhosting system. With so may options, another may be the best choice for your specific needs. Consider investigating
The following packages are used in this setup: apache, courier-imap, pam_mysql, postfix, mod_php, mod_ssl, phpmyadmin, squirrelmail, cyrus-sasl, mysql, php, and mailman.
Make sure to turn on the following USE variables in
# emerge postfix
After postfix is installed, it's time to configure it. Change the following options in
myhostname = $host.domain.name mydomain = $domain.name myorigin = $mydomain inet_interfaces = all mydestination = $myhostname, localhost.$mydomain $mydomain relay_domains = $mydestination mynetworks = my.ip.net.work/24, 127.0.0.0/8 local_destination_concurrency_limit = 2 default_destination_concurrency_limit = 10
Next change the following in
# service type private unpriv chroot wakeup maxproc command + args # (yes) (yes) (yes) (never) (50) # ========================================================================== smtp inet n - n - - smtpd -vJust add the -v after the smtpd in the above line
Next, edit
# /usr/bin/newaliasesThis will install the new aliases. You only need to do this when you update or install aliases. # /etc/init.d/postfix start
Now that postfix is running, fire up your favorite console mail client and send yourself an email. I use
# emerge courier-imap
# cd /etc/courier-imapIf you want to use the ssl capabilities of courier-imap or pop3, you'll need to create certs for this purpose. This step is recommended. If you do not want to use ssl, skip this step. # nano pop3d.cnf # nano imapd.cnfChange the C, ST, L, CN, and email parameters to match your server. # mkpop3dcert # mkimapdcert
# courier-imapd start # courier-imapd-ssl start # courier-pop3d start # courier-pop3d-ssl start
Start up your favorite mail client and verify that all connections you've started work for receiving and sending mail. Now that the basics work, we're going to do a whole bunch of stuff at once to get the rest of the system running. Again, please verify that what we've installed already works before progressing.
Next we're going to install cyrus-sasl. Sasl is going to play the role of actually passing your auth variables to pam, which will in turn pass that information to mysql for authentication of smtp users. For this howto, we'll not even try to verify that sasl is working until mysql is set up and contains a test user. Which is fine since we'll be authenticating against mysql in the end anyway.
Just to get sasl installed is going to require a bit of hacking. Open up the ebuild file and change the configure flags to disable digest and cram. Here's why: mail clients will try to authenticate against the
# cd /usr/portage/dev-libs/cyrus-sasl # nano cyrus-sasl.$currentversion.ebuildDisable digest and cram as show below. econf \ --with-saslauthd=/var/lib/sasl2 \ --with-pwcheck=/var/lib/sasl2 \ --with-configdir=/etc/sasl2 \ --with-openssl \ --with-plugindir=/usr/lib/sasl2 \ --with-dbpath=/etc/sasl2/sasldb2 \ --with-des \ --with-rc4 \ --disable-krb4 \ --with-gnu-ld \ --enable-shared \ --disable-sample \ --enable-login \ --disable-cram \ --disable-digest \ ${myconf} || die "bad ./configure" # USE='-ldap -mysql' emerge cyrus-saslWe don't have ldap and we're not using sasl's mysql capabilities so we need to turn them off for this build.
Now check that the directory
# mkdir /var/lib/sasl2If saslauthd can't find the dir it will spew errors at you and refuse to start. # /etc/init.d/saslauthd start
Next we're going to make a set of ssl certificates for postfix and apache.
# cd /usr/lib/ssl/ # nano openssl.cnfChange the following default values for your domain: countryName_default stateOrProvinceName_default localityName_default 0.organizationName_default commonName_default emailAddress_default.If the variables are not already present, just add them in a sensible place. # cd misc # nano CA.plWe need to add -nodes to the # create a certificate and #create a certificate request code in order to let our new ssl certs be loaded without a password. Otherwise when you reboot your ssl certs will not be available. # create a certificate system ("$REQ -new -nodes -x509 -keyout newreq.pem -out newreq.pem $DAYS"); # create a certificate request system ("$REQ -new -nodes -keyout newreq.pem -out newreq.pem $DAYS"); # ./CA.pl -newcert # ./CA.pl -newreq # ./CA.pl -sign # cp newcert.pem /etc/postfix # cp newreq.pem /etc/postfix # cp demoCA/cacert.pem /etc/postfixNow we do the same thing for apache # openssl req -new > new.cert.csr # openssl rsa -in privkey.pem -out new.cert.key # openssl x509 -in new.cert.csr -out new.cert.cert -req -signkey new.cert.key -days 365Just leave the resulting certificates here for now. We'll install them after Apache is installed.
Now edit the postfix config's to make it aware of your new sasl and ssl capabilities. Add the following parameters to the end of the file where they will be easy to find.
# nano /etc/postfix/main.cf smtpd_sasl_auth_enable = yes smtpd_sasl2_auth_enable = yes smtpd_sasl_security_options = noanonymous broken_sasl_auth_clients = yesThe broken_sasl_auth_clients option and the login auth method are for outlook and outlook express only and are undocumented. Isn't having to hack software for stupid, broken, M$ BS great? smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, check_relay_domains, permit smtpd_use_tls = yes #smtpd_tls_auth_only = yes smtpd_tls_key_file = /etc/postfix/newreq.pem smtpd_tls_cert_file = /etc/postfix/newcert.pem smtpd_tls_CAfile = /etc/postfix/cacert.pem smtpd_tls_loglevel = 3 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s tls_random_source = dev:/dev/urandomsmtpd_tls_auth_only is commented out to ease testing the system. You can turn this on later if you desire. # postfix reload
Now we're going to verify that the config's we added were picked up by postfix.
# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 mail.domain.com ESMTP Postfix EHLO domain.com 250-mail.domain.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-STARTTLS 250-AUTH LOGIN PLAIN OTP 250-AUTH=LOGIN PLAIN OTP 250-XVERP 250 8BITMIME ^] telnet> quit
Verify that the above AUTH and STARTTLS lines now appear in your postfix install. As I said before, as it stands now AUTH will not work. that's because sasl will try to auth against it's sasldb, instead of the shadow file for some unknown reason, which we have not set up. So we're going to just plow through and set up mysql to hold all of our auth and virtual domain information.
Next we're going to install and configure MySQL. You'll need the
# emerge mysql # /usr/sbin/mysql_install_dbAfter this command runs follow the onscreen directions for adding a root password with mysql, not mysqladmin, otherwise your db will be wide open. # /etc/init.d/mysql start # mysqladmin -u root -p create mailsql # mysql -u root -p --opt mailsql < genericmailsql.sql # mysql -u root -p mysql mysql> GRANT SELECT,INSERT,UPDATE,DELETE -> ON mailsql.* -> TO mailsql@localhost -> IDENTIFIED BY '$password'; -> quitVerify that the new mailsql user can connect to the mysql server. # mysql -u mailsql -p mailsql
Your new database has default values and tables set up for two domains. The following tables are included:
Next we'll set up apache and add an interface to interact with the database more easily.
# emerge apache php mod_php mod_ssl phpmyadmin
There are plenty of guides out there about how to set up apache with php. Like this one:
Now we're going to install the Apache certificates we made previously. The Apache-SSL directives that you need to use the resulting cert are:
# cp /usr/lib/ssl/misc/new.cert.cert /etc/apache/conf/ssl/ # cp /usr/lib/ssl/misc/new.cert.key /etc/apache/conf/ssl/ # nano /etc/apache/conf/vhosts/ssl.default-vhost.confChange the following parameters ServerName host.domain.name ServerAdmin your@email.address SSLCertificateFile /etc/apache/conf/ssl/new.cert.cert SSLCertificateKeyFile /etc/apache/conf/ssl/new.cert.key
Next, configure phpMyAdmin.
# nano /home/httpd/htdocs/phpmyadmin/config.inc.phpChange the following parameters $cfg['Servers'][$i]['host'] = 'localhost'; // MySQL hostname $cfg['Servers'][$i]['controluser'] = 'mailsql'; // MySQL control user settings // (this user must have read-only $cfg['Servers'][$i]['controlpass'] = '$password'; // access to the "mysql/user" // and "mysql/db" tables) $cfg['Servers'][$i]['user'] = 'mailsql'; // MySQL user $cfg['Servers'][$i]['password'] = '$password'; // MySQL password
Now enter the phpmyadmin page and browse the tables. You'll want to add in your local aliases, edit your user table to add a test user, and change your transport table to add information about your domains. The default values supplied with the dumpfile should be a sufficient guide to what values need to go where. Make sure that if you put information in the database that it is accurate. For instance, make sure the local users home dir exists and that the correct uid/gid values are in place. The maildirs should be created automatically by postfix when the user receives their first email. So, in general, it's a good idea to send a "Welcome" mail to a new user after you setup their account to make sure the .maildir gets created.
At this point you may be wondering what user and directory to use for virtual mail users, and rightly so. Let's set that up.
# adduser -d /home/vmail -s /bin/false vmail # uid=`cat /etc/passwd | grep vmail | cut -f 3 -d :` # groupadd -g $uid vmail # mkdir /home/vmail # chown vmail. /home/vmail
So now when you're setting up vmail accounts, use the vmail uid, gid, and homedir. When you're setting up local accounts, use that users uid, gid, and homedir. We've been meaning to create a php admin page for this setup but haven't gotten around to it yet, as phpmyadmin generally works fine for us.
Next we'll reconfigure our authentication to use the mailsql database in courier-imap and postfix. In all of the following examples, replace $password with the password you set for the mailsql mysql user.
# emerge /usr/portage/sys-libs/pam_mysql/pam_mysql-$currentversion.ebuildThis package is currently masked so you'll have to give it an explicit path or unmask the package. This is subject to change when the package is unmasked in the portage tree. # nano /etc/pam.d/imapComment out the existing auth lines and add the following as shown. #auth required pam_nologin.so #auth required pam_stack.so service=system-auth #account required pam_stack.so service=system-auth #session required pam_stack.so service=system-auth auth optional pam_mysql.so server=localhost db=mailsql user=mailsql \ passwd=$password table=users usercolumn=email passwdcolumn=clear crypt=n account required pam_mysql.so server=localhost db=mailsql user=mailsql \ passwd=$password table=users usercolumn=email passwdcolumn=clear crypt=n # nano /etc/pam.d/pop3 # nano /etc/pam.d/smtpMake the same changes to the pop3 and smtp files
Next, we need to edit courier's authentication config's.
# nano /etc/courier-imap/authdaemonrc authmodulelist="authmysql authpam" # nano /etc/courier-imap/authdaemond.conf AUTHDAEMOND="authdaemond.mysql" # nano /etc/courier-imap/authmysqlrc MYSQL_SERVER localhost MYSQL_USERNAME mailsql MYSQL_PASSWORD $password MYSQL_DATABASE mailsql MYSQL_USER_TABLE users #MYSQL_CRYPT_PWFIELD crypt (make sure this is commented out since we're storing plaintext) MYSQL_CLEAR_PWFIELD clear MYSQL_UID_FIELD uid MYSQL_GID_FIELD gid MYSQL_LOGIN_FIELD email MYSQL_HOME_FIELD homedir MYSQL_NAME_FIELD name MYSQL_MAILDIR_FIELD maildir # /etc/init.d/authdaemond restart # /etc/init.d/saslauthd restart
We're almost there I promise! Next, set up the rest of the necessary config's for postfix to interract with the database for all it's other transport needs.
# nano /etc/postfix/mysql-aliases.cf # mysql-aliases.cf user = mailsql password = $password dbname = mailsql table = alias select_field = destination where_field = alias hosts = 127.0.0.1
# nano /etc/postfix/mysql-relocated.cf # mysql-relocated.cf user = mailsql password = $password dbname = mailsql table = relocated select_field = destination where_field = email hosts = 127.0.0.1
# nano /etc/postfix/mysql-transport.cf # mysql-transport.cf user = mailsql password = $password dbname = mailsql table = transport select_field = destination where_field = domain hosts = 127.0.0.1
# nano /etc/postfix/mysql-virtual-gid.cf #myql-virtual-gid.cf user = mailsql password = $password dbname = mailsql table = users select_field = gid where_field = email additional_conditions = and postfix = 'y' hosts = 127.0.0.1
# nano /etc/postfix/mysql-virtual-maps.cf #myql-virtual-maps.cf user = mailsql password = $password dbname = mailsql table = users select_field = maildir where_field = email additional_conditions = and postfix = 'y' hosts = 127.0.0.1
# nano /etc/postfix/mysql-virtual-uid.cf # mysql-virtual-uid.cf user = mailsql password = $password dbname = mailsql table = users select_field = uid where_field = email additional_conditions = and postfix = 'y' hosts = 127.0.0.1
# nano /etc/postfix/mysql-virtual.cf # mysql-virtual.cf user = mailsql password = $password dbname = mailsql table = virtual select_field = destination where_field = email hosts = 127.0.0.1
Lastly, edit
# nano /etc/postfix/main.cf alias_maps = mysql:/etc/postfix/mysql-aliases.cf relocated_maps = mysql:/etc/postfix/mysql-relocated.cf transport_maps = mysql:/etc/postfix/mysql-transport.cf virtual_gid_maps = mysql:/etc/postfix/mysql-virtual-gid.cf virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-maps.cf virtual_maps = mysql:/etc/postfix/mysql-virtual.cf virtual_uid_maps = mysql:/etc/postfix/mysql-virtual-uid.cf
Shew! Alright now, let's restart postfix and see how all that went.
# postfix reload
Now, if all went well, you should have a functioning mailhost. Users should be able to authenticate against the sql database, using their full email address, for pop3, imap, and smtp. I would highly suggest that you verify that everything is working at this point. If you run into problems (with as many things as this setup has going on, it's likely that you will) check the troubleshooting section of this howto.
# emerge squirrelmailI like to add a link to the htdocs space for a shorter url. # ln -s /home/httpd/htdocs/squirrelmail/ /home/httpd/htdocs/mail # cd /home/httpd/htdocs/mail/conf # ./conf.plChange your Organization, Server, and Folder settings for squirrelmail. Now you should be able to login to squirrelmail, again - with your full email address, and use your new webmail setup.
Last step: mailman. The new version of mailman has very nice virtual domain support, which is why I use it, not to mention it's really a great package.
# emerge mailmanThis package is currently masked as well, so you'll need to unmask it or give emerge an explicit path to the ebuild. Once it's installed, follow the directions in the README.gentoo # nano /usr/share/doc/mailman-$ver/README.gentooFollow the rest of the install instructions in the README. Once that's finished, add your first list. # su mailman # cd ~ # bin/newlist
That script should spit out some aliases at you for postfix. Add those aliases to the alias table in mailsql. You'll be tempted to add colons to the alias names in mailsql (because that's how the script gives them to you) DON'T DO IT! It won't work if you do. You should now be able to setup mailing lists for any domain on your box. When you log in to the mailman admin page for your new list, at the bottom of the general options page there is a space to declare what domain the list belongs to. With that selection you can add lists for any virtual domain you are hosting. Last note on this, make sure you run all mailman commands as the user mailman (
Ok, you're all set, edit
# postfix reload # rc-update add $service default
Troubleshooting: This is a short troubleshooting guide for the set up we've detailed how to install here. It is not exhaustive, but meant as a place to get you started in figuring out problems. With a complicated setup such as this, it's imperative that you narrow down the problem to the particular component that is malfunctioning. In general I do that by following a few steps. Start from the base of the system and work your way up, ruling out components that work along the way until you discover which component is having the problem.
Typos are killers, especially when dealing with authentication systems. Scan your config's and mailsql database for typo's. You can debug all you want, but if you're not passing the right information back and forth to your mail system, it's not going to work. If you make a change to a config file for a service, make sure you restart that service so that the config change gets picked up.
# /etc/init.d/service restart
If it's not running, start it up. It's awful hard to debug a service that isn't running. Sometimes a service will act like it's started but still not function. Sometimes, when a bad config is used, or a bad transmission comes into a mail component, the service will hang and keep the port from being used by another process. Sometimes you can detect this with netstat. Or, if you've been at it awhile, just take a break and reboot your box in the meantime. That will clear out any hung services. Then you can come back fresh and try it again.
# /etc/init.d/$service status # netstat -a | grep $service (or $port)
If you've recently made a change to a config file, restart that service to make sure it's using the current version. Some of the components will dump their current config's to you, like postfix.
# apachectl fullstatus (needs lynx installed) # apachectl configtest (checks config sanity) # postconf (will tell you exactly what param's postfix is using) # /etc/init.d/$service restart
Repeat after me, logs are my friend. My next troubleshooting stop is always the logs. Sometimes it's helpful to try a failed operation again then check the logs so that the error message is right at the bottom (or top depending on your logger) instead of buried in there somewhere. See if there is any information in your log that can help you diagnose the problem, or at the very least, figure out which component is having the problem.
# kill -USR1 `ps -C metalog -o pid=`(to turn off metalog buffering) # nano /var/log/everything/current # cat /var/log/mysql/mysql.log # tail /var/log/apache/error_log
SMTP, IMAP, and POP3 all respond to telnet sessions. As we've seen earlier when we verified postfix's config. Sometimes it's helpful to open a telnet session to the service itself and see what's happening.
# telnet localhost $portSMTP is 25, IMAP is 143, POP3 is 110. You should receive at least an OK string, letting you know that the service is running and ready to respond to requests. Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK Courier-IMAP ready. Copyright 1998-2002 Double Precision, Inc. See COPYING for distribution information.
You should have this installed anyway. This is an invaluable tool for debugging software. You can start commands from the command line with strace and watch all the system calls as they happen. It often dumps a huge amount of information, so you'll either need to watch it realtime as you retry a failed transaction with the mail system, or dump the output to a file for review.
# emerge strace # strace $command # strace -p `ps -C $service -o pid=`
Once you have the information, if you can diagnose and fix the problem, great! If not, you'll probably need to go digging on the net for information that will help you fix it. Here's a list of sites you can check to see if your error has already been resolved. There's also a really good howto on setting up smtp-auth which contains some great debugging ideas.