For some time now, I was experiencing a strange behavior of my server: from time to time, without an ascertainable pattern, the server would stop reacting to network-requests. The teamspeak-server, which runs on it, would kick anyone connected to it, and nothing particular special could be found in the logs. When this happened last Thursday, and I was kicked out of the teamspeak-server myself, I tried to ssh onto my server - which took about 30 seconds. This was irritating, and I run "uptime" to check the server's load - it was way beyond 70. The next thing was a call to top, and here I saw the culprit: sendmail. A call to ps verified the sendmail was running with way too many processes, all in "RCPT TO:" state or something similar. I stopped sendmail and killed remaining processes manually, so that I could work again in real time. Looking through maillog, I began to understand what going on: spammers were DDoSing my mail-server. Though I already had some settings in my sendmail.mc that would make the server unattractive for spammers, they were obviously not sufficient, and especially not against DDoS-attacks. So I changed my configuration a bit.
Connection Controlling
FEATURE(`access_db')dnl FEATURE(`delay_checks', `friend')dnl FEATURE(`ratecontrol', `nodelay',`terminate')dnl FEATURE(`conncontrol', `nodelay',`terminate')dnl
After the already existing line "FEATURE(`access_db')dnl" I added the lines to enable rate and connection-controlling.
The option "nodelay" is important, because I am using the delay-checks-feature, and these checks are not to be delayed.
The "terminate"-option tells sendmail to kill all connections exceeding the later defined limits with a temporary error-message. The properly configured and standard-compliant smtp-client will try again later, spammers usually don't.
The rate-control feature enables control over how often a single host is allowed to connect per a defined window. It was introduce by sendmail version 8.13.0, and uses the access.db for defining the limits for single hosts, groups of hosts or all hosts.
The window-size is defined using the this option:
define(`confCONNECTION_RATE_WINDOW_SIZE',`window')dnl
with a default-value for window of "60s" (60 seconds).
My access.db-entries for the rate-control-feature look like this:
ClientRate:localhost 0 ClientRate:localhost.localdomain 0 ClientRate:127.0.0.1 0 ClientRate: 5
The first three lines tell sendmail to ingore rate-limits for the localhost, and the last line imposes a limit of 5 connections per window for all hosts.
In cases of a DDoS, it might not be sufficient to limit the connections of a single host per minute, because a DDoS comes from multiple hosts at the same time. This is why Sendmail come with another option:
define(`confCONNECTION_RATE_THROTTLE', `5')dnl
This defines the overall number of concurrent connection the server accepts per second, before queuing incoming connection-request regardless of the host. The connections will not be rejected but stalled until the next second. This means that for the above example that when 20 connection-requests arrive, the first five (1-5) are processed in second one, the second five (6-10) in second two, the third five (11-15) in second three, and the final five (16-20) in second four.
The conn-control feature enables control over the number of concurrent connections a single host is allowed to run simultaneously. Like rate-control, this feature was introduced with Sendmail 8.13, and the access.db is used to define settings for single-hosts, host-ranges and "all hosts", too.
My access.db-entries for the conn-control-feature look like this:
ClientConn:localhost 0 ClientConn:localhost.localdomain 0 ClientConn:127.0.0.1 0 ClientConn: 3
The entries are read similar to rate-control. The last line defines a default of 3 concurrent connections, the first three disable the feature for localhost.
Greeting Pause
A common technique of spammers, trojans and viruses is the so-called slamming. The SMTP-Standard requires the client to wait with the HELO/EHLO-Command until the server has sent its greeting line. Slamming is to ignore this, and to start sending immediately.
FEATURE(`greet_pause', `2000')dnl
With the above feature, Sendmail can be configured to delay the sending of this greeting. The value is in milliseconds, so in the example above, the greeting-pause would be two seconds. A client issuing the HELO/EHLO during this pause will cause Sendmail to answer with
554 smtp.nifelhei.info not accepting messages
and the greeting will not be send. Sendmail will log such attempts with a message like
rejecting messages from <host> due to pre-greeting traffic.
and terminate the connection.
You can use the access.db again to define host-specific greeting-pause times, or to exclude certain hosts from the pause. The following example would exclude localhost from the delay. You can use this to whitelist smtp-servers who do slamming but are otherwise "friendly".
GreetPause:localhost 0 GreetPause:localhost.localdomain 0 GreetPause:127.0.0.1 0
Please note:
RFC 2821 specifies 5 minutes as the maximum timeout for the initial connection greeting. Therefore, if you specify a time longer than 300000 milliseconds (i.e. 5 minutes), sendmail will not wait longer than 5 minutes, to maintain RFC compliance.
Recipient-Controlling
After setting up the controlling mechanisms for incoming connections, there is a another level of control that can be applied. Many spammers try to send a single mail with hundreds of recipients. This is also known as "recipient flooding". Sendmail can be configured to limit the number of recipients a message may have, as well throttling down all those clients who try to add more recipient than a certain threshold by pausing a hardcoded full second between each accepted recipient. The options are as follows:
define(`confBAD_RCPT_THROTTLE', `2')dnl define(`confMAX_RCPTS_PER_MESSAGE', `25')dnl
BAD_RCPT_THROTTLE sets the threshold which invokes the one-second-delay. For the example above this means that with the third RCPT TO: sendmail will pause one full second, before sending the response.
MAX_RCPTS_PER_MESSAGE limits the absolute maximum number of recipients for each message to the value given (25 for the above example). Every RCPT TO: exceeding this number will be rejected with an appropriate message. The standard-compliant server will collect the rejected RCPT TOs and requeue the message for all yet outstanding recipients. (Yes, spammers won't.)
Timeouts
Sendmail, in order to get as many as possible mails through, has very generous timeout-defaults. These values are often measured in days, where today seconds or minutes would suffice. Long timeouts mean long bound resources for probably unsolicited connections. I have defined much shorter values for several timeouts:
define(`confTO_INITIAL', `30s')dnl define(`confTO_CONNECT', `30s')dnl define(`confTO_ACONNECT', `1m')dnl define(`confTO_ICONNECT', `30s')dnl define(`confTO_HELO', `30s')dnl define(`confTO_MAIL', `30s')dnl define(`confTO_RCPT', `30s')dnl define(`confTO_DATAINIT', `1m')dnl define(`confTO_DATABLOCK', `1m')dnl define(`confTO_DATAFINAL', `1m')dnl define(`confTO_RSET', `30s')dnl define(`confTO_QUIT', `30s')dnl define(`confTO_MISC', `30s')dnl define(`confTO_COMMAND', `30s')dnl define(`confTO_CONTROL', `30s')dnl define(`confTO_LHLO', `30s')dnl define(`confTO_AUTH', `30s')dnl define(`confTO_STARTTLS', `30s')dnl
I won't go into much detail about each timeout, because that would be beyond the scope of this posting, but these values are much more reasonable than the defaults.
Other means of protecting your server agains spammers:
- DNS blacklisting
- Greylisting
- Spam-filtering using external filters like SpamAssassin or MIMEDefang
- TCP Wrappers for blacklisting known malicious hosts/subnets, subsection "TCP Wrappers" below.
TCPWrappers
Besides everything sendmail can be configured to do and not to do, sendmail has another advantage: It can be compiled to use TCP Wrapper.
While scanning the logs for the causes of the astronomous load, I noticed millions of attempts from hosts of dial-in providers, which usually strongly indicates spam-bot afflicted private hosts.
I have added theses networks to my /etc/hosts.deny file, with the effect that the number of connections to the server was reduced almost immediately. While one might ask for the wisdom of blocking whole networks, think about this: by what necessity does a private dial-in host have to have its own smtp-server attempting to connect to your smtp-server? Usually, a private person can use the mail-server of his/her provider, and that one won't be blocked, because I am blocking the dial-in-subnets specifically.
Here is the current (March 29, 2009) list of blocked networks:
sendmail: .adsl.alicedsl.de sendmail: .tukw.qwest.net sendmail: .internetdsl.tpnet.pl sendmail: .dynamicIP.rima-tde.net sendmail: .staticIP.rima-tde.net sendmail: .home.otenet.gr sendmail: .pppoe.mtu-net.ru sendmail: .static.link.com.eg sendmail: .adsl-1.sezampro.yu sendmail: .speedy.telkom.net.id sendmail: .pool.ukrtel.net sendmail: .taiwanmobile.net sendmail: .veloxzone.com.br sendmail: .bielskpodlaski.mm.pl sendmail: .bb-static.vsnl.net.in sendmail: .dynamic.163data.com.cn sendmail: .vsnl.net.in sendmail: .adsl.tpnet.pl sendmail: .airtelbroadband.in sendmail: .ip.adsl.hu sendmail: .tktelekom.pl sendmail: .radiocom.ro sendmail: .static.asianet.co.th sendmail: .static.versatel.nl sendmail: .dsl.telesp.net.br sendmail: .cable.telstraclear.net sendmail: .bb.netvision.net.il sendmail: .ip.fastwebnet.it sendmail: .pppoe.avangarddsl.ru sendmail: .adsl.proxad.net sendmail: .adsl.sta.mcn.ru sendmail: .adsl.paltel.net sendmail: .iam.net.ma sendmail: .mobile.playmobile.pl sendmail: .broadband3.iol.cz sendmail: .business.telecomitalia.it sendmail: .sonora.tx.cebridge.net sendmail: .3g.claro.net.br sendmail: .wi.res.rr.com sendmail: .mtnl.net.in sendmail: .static.gvt.net.br sendmail: .dynamic.orange.es sendmail: .ttnet.net.tr sendmail: .ip.cybergrota.com.pl sendmail: .static.user.ono.com sendmail: .dsl.brasiltelecom.net.br sendmail: .bk21-dsl.surnet.cl
Conclusion
After changing the configuration using the above described possibilities, the load of the sever decreased enormously, and there are far less sendmail-processes now running at the same time, thus binding far less resources. DDoS-spam-attacks are still not impossible, but they will have a harder time to get the machine down now. 
Tags: Configuration, DDoS, denial-of-service, hardening, mail-server, security, sendmail
I recently came across your blog and have been reading along. I thought I would leave my first comment. I was searching regarding mail servers and landed here. I don’t know what to say except that I have enjoyed reading. This blog is very informative . Thanks for the post.
[Reply]
Thanks for this article.
But in my case, the main hungry proccess is the antivir (ClamAv). Running “top” command, and ordering by CPU usage, I can realise that ClamAv is the main “problem”.
I use today’s latest version (ClamAV 0.95.2), with the lastest version of MailScanner.
Our server process 25K mails per day, and 98% are Spam. The load average of the server is bettwen 1 - 2.5. Do you know how to reduce this high load average ?
I think it could be enhaced, running clamav as a daemon (clamd), instead of a process on the fly.
What do you think ?
[Reply]
mjoellnir reply on October 9th, 2009:
Hello Juan,
yes, indeed, you should run clamd and tell MailScanner to communicate through unix- or tcp-sockets with clamd. Using ClamAV like you do would mean several instances of ClamAV with each loading the complete database. That is quite resource-consuming, and slow as well. If instead there is clamd running, and MailScanner (or as in my case, clamav-milter), this problem will gone immediately.
I would generally advise the usage of clamd for every mailserver (using ClamAV), by the way, no matter how many mails are processed per day. It’s the more efficient way of handling these things.
[Reply]
joe reply on February 6th, 2010:
If you need your clamav installation to perform better, try put the scanning in a ramdisk filesystem.
[Reply]