Posts Tagged ‘denial-of-service’

Hardening Sendmail - supplement

Tuesday, March 31st, 2009

In my last post I wrote about hardening sendmail against DDoS-Attacks. As someone has pointed out to me, I have missed an important option:

define(`confMAX_DAEMON_CHILDREN', `count')dnl

This option defines the maximum number of sendmail-processes allowed, before sendmail start rejecting incoming connections with a temporary error.

count should be chosen with great care. I recommend to check the average number of sendmail-processes per second on a "standard" day, and triple that number. This way, you ensure that even in peak-times you will have a high enough limit, but in case of real trouble the number of processes won't explode, and the machine will remain operable.

Example: If you have an average of 20 concurrent sendmail-processes, set count to 60.  I would never recommend a value below 30, though.

Hardening Sendmail against DDoS

Sunday, March 29th, 2009

For some time now, I was experiencing a strange behavior of my server: from time to time, without an ascertainable pattern, the server would stop reacting to network-requests. The teamspeak-server, which runs on it, would kick anyone connected to it, and nothing particular special could be found in the logs. When this happened last Thursday, and I was kicked out of the teamspeak-server myself, I tried to ssh onto my server - which took about 30 seconds. This was irritating, and I run "uptime" to check the server's load - it was way beyond 70. The next thing was a call to top, and here I saw the culprit: sendmail. A call to ps verified the sendmail was running with way too many processes, all in "RCPT TO:" state or something similar. I stopped sendmail and killed remaining processes manually, so that I could work again in real time. Looking through maillog, I began to understand what going on: spammers were DDoSing my mail-server. Though I already had some settings in my sendmail.mc that would make the server unattractive for spammers, they were obviously not sufficient, and especially not against DDoS-attacks. So I changed my configuration a bit.

Connection Controlling

FEATURE(`access_db')dnl
FEATURE(`delay_checks', `friend')dnl
FEATURE(`ratecontrol', `nodelay',`terminate')dnl
FEATURE(`conncontrol', `nodelay',`terminate')dnl

After the already existing line "FEATURE(`access_db')dnl" I added the lines to enable rate and connection-controlling.

The option "nodelay" is important, because I am using the delay-checks-feature, and these checks are not to be delayed.

The "terminate"-option tells sendmail to kill all connections exceeding the later defined limits with a temporary error-message. The properly configured and standard-compliant smtp-client will try again later, spammers usually don't.

The rate-control feature enables control over how often a single host is allowed to connect per a defined window. It was introduce by sendmail version 8.13.0, and uses the access.db for defining the limits for single hosts, groups of hosts or all hosts.

The window-size is defined using the this option:

define(`confCONNECTION_RATE_WINDOW_SIZE',`window')dnl

with a default-value for window of "60s" (60 seconds).

My access.db-entries for the rate-control-feature look like this:

ClientRate:localhost                    0
ClientRate:localhost.localdomain        0
ClientRate:127.0.0.1                    0
ClientRate:                             5

The first three lines tell sendmail to ingore rate-limits for the localhost, and the last line imposes a limit of 5 connections per window for all hosts.

In cases of a DDoS, it might not be sufficient to limit the connections of a single host per minute, because a DDoS comes from multiple hosts at the same time. This is why Sendmail come with another option:

define(`confCONNECTION_RATE_THROTTLE', `5')dnl

This defines the overall number of concurrent connection the server accepts per second, before queuing incoming connection-request regardless of the host. The connections will not be rejected but stalled until the next second. This means that for the above example that when 20 connection-requests arrive, the first five (1-5) are processed in second one, the second five (6-10) in second two, the third five (11-15) in second three, and the final five (16-20) in second four.

The conn-control feature enables control over the number of concurrent connections a single host is allowed to run simultaneously. Like rate-control, this feature was introduced with Sendmail 8.13, and the access.db is used to define settings for single-hosts, host-ranges and "all hosts", too.

My access.db-entries for the conn-control-feature look like this:

ClientConn:localhost                    0
ClientConn:localhost.localdomain        0
ClientConn:127.0.0.1                    0
ClientConn:                             3

The entries are read similar to rate-control. The last line defines a default of 3 concurrent connections, the first three disable the feature for localhost.

Greeting Pause

A common technique of spammers, trojans and viruses is the so-called slamming. The SMTP-Standard requires the client to wait with the HELO/EHLO-Command until the server has sent its greeting line. Slamming is to ignore this, and to start sending immediately.

FEATURE(`greet_pause', `2000')dnl

With the above feature, Sendmail can be configured to delay the sending of this greeting. The value is in milliseconds, so in the example above, the greeting-pause would be two seconds. A client issuing the HELO/EHLO during this pause will cause Sendmail to answer with

554 smtp.nifelhei.info not accepting messages

and the greeting will not be send. Sendmail will log such attempts with a message like

rejecting messages from <host> due to pre-greeting traffic.

and terminate the connection.

You can use the access.db again to define host-specific greeting-pause times, or to exclude certain hosts from the pause. The following example would exclude localhost from the delay. You can use this to whitelist smtp-servers who do slamming but are otherwise "friendly".

GreetPause:localhost                    0
GreetPause:localhost.localdomain        0
GreetPause:127.0.0.1                    0

Please note:

RFC 2821 specifies 5 minutes as the maximum timeout for the initial connection greeting. Therefore, if you specify a time longer than 300000 milliseconds (i.e. 5 minutes), sendmail will not wait longer than 5 minutes, to maintain RFC compliance.

Recipient-Controlling

After setting up the controlling mechanisms for incoming connections, there is a another level of control that can be applied. Many spammers try to send a single mail with hundreds of recipients. This is also known as "recipient flooding". Sendmail can be configured to limit the number of recipients a message may have, as well throttling down all those clients who try to add more recipient than a certain threshold by pausing a hardcoded full second between each accepted recipient. The options are as follows:

define(`confBAD_RCPT_THROTTLE', `2')dnl
define(`confMAX_RCPTS_PER_MESSAGE', `25')dnl

BAD_RCPT_THROTTLE sets the threshold which invokes the one-second-delay. For the example above this means that with the third RCPT TO: sendmail will pause one full second, before sending the response.

MAX_RCPTS_PER_MESSAGE limits the absolute maximum number of recipients for each message to the value given (25 for the above example). Every RCPT TO: exceeding this number will be rejected with an appropriate message. The standard-compliant server will collect the rejected RCPT TOs and requeue the message for all yet outstanding recipients. (Yes, spammers won't.)

Timeouts

Sendmail, in order to get as many as possible mails through, has very generous timeout-defaults. These values are often measured in days, where today seconds or minutes would suffice. Long timeouts mean long bound resources for probably unsolicited connections. I have defined much shorter values for several timeouts:

define(`confTO_INITIAL', `30s')dnl
define(`confTO_CONNECT', `30s')dnl
define(`confTO_ACONNECT', `1m')dnl
define(`confTO_ICONNECT', `30s')dnl
define(`confTO_HELO', `30s')dnl
define(`confTO_MAIL', `30s')dnl
define(`confTO_RCPT', `30s')dnl
define(`confTO_DATAINIT', `1m')dnl
define(`confTO_DATABLOCK', `1m')dnl
define(`confTO_DATAFINAL', `1m')dnl
define(`confTO_RSET', `30s')dnl
define(`confTO_QUIT', `30s')dnl
define(`confTO_MISC', `30s')dnl
define(`confTO_COMMAND', `30s')dnl
define(`confTO_CONTROL', `30s')dnl
define(`confTO_LHLO', `30s')dnl
define(`confTO_AUTH', `30s')dnl
define(`confTO_STARTTLS', `30s')dnl

I won't go into much detail about each timeout, because that would be beyond the scope of this posting, but these values are much more reasonable than the defaults.

Other means of protecting your server agains spammers:

TCPWrappers

Besides everything sendmail can be configured to do and not to do, sendmail has another advantage: It can be compiled to use TCP Wrapper.

While scanning the logs for the causes of the astronomous load, I noticed millions of attempts from hosts of dial-in providers, which usually strongly indicates spam-bot afflicted private hosts.

I have added theses networks to my /etc/hosts.deny file, with the effect that the number of connections to the server was reduced almost immediately. While one might ask for the wisdom of blocking whole networks, think about this: by what necessity does a private dial-in host have to have its own smtp-server attempting to connect to your smtp-server? Usually, a private person can use the mail-server of his/her provider,  and that one won't be blocked, because I am blocking the dial-in-subnets specifically.

Here is the current (March 29, 2009) list of blocked networks:

sendmail: .adsl.alicedsl.de
sendmail: .tukw.qwest.net
sendmail: .internetdsl.tpnet.pl
sendmail: .dynamicIP.rima-tde.net
sendmail: .staticIP.rima-tde.net
sendmail: .home.otenet.gr
sendmail: .pppoe.mtu-net.ru
sendmail: .static.link.com.eg
sendmail: .adsl-1.sezampro.yu
sendmail: .speedy.telkom.net.id
sendmail: .pool.ukrtel.net
sendmail: .taiwanmobile.net
sendmail: .veloxzone.com.br
sendmail: .bielskpodlaski.mm.pl
sendmail: .bb-static.vsnl.net.in
sendmail: .dynamic.163data.com.cn
sendmail: .vsnl.net.in
sendmail: .adsl.tpnet.pl
sendmail: .airtelbroadband.in
sendmail: .ip.adsl.hu
sendmail: .tktelekom.pl
sendmail: .radiocom.ro
sendmail: .static.asianet.co.th
sendmail: .static.versatel.nl
sendmail: .dsl.telesp.net.br
sendmail: .cable.telstraclear.net
sendmail: .bb.netvision.net.il
sendmail: .ip.fastwebnet.it
sendmail: .pppoe.avangarddsl.ru
sendmail: .adsl.proxad.net
sendmail: .adsl.sta.mcn.ru
sendmail: .adsl.paltel.net
sendmail: .iam.net.ma
sendmail: .mobile.playmobile.pl
sendmail: .broadband3.iol.cz
sendmail: .business.telecomitalia.it
sendmail: .sonora.tx.cebridge.net
sendmail: .3g.claro.net.br
sendmail: .wi.res.rr.com
sendmail: .mtnl.net.in
sendmail: .static.gvt.net.br
sendmail: .dynamic.orange.es
sendmail: .ttnet.net.tr
sendmail: .ip.cybergrota.com.pl
sendmail: .static.user.ono.com
sendmail: .dsl.brasiltelecom.net.br
sendmail: .bk21-dsl.surnet.cl

Conclusion

After changing the configuration using the above described possibilities, the load of the sever decreased enormously, and there are far less sendmail-processes now running at the same time, thus binding far less resources. DDoS-spam-attacks are still not impossible, but they will have a harder time to get the machine down now. :)

lighttpd <= 1.4.19 has a denial of service vulnerability

Monday, April 7th, 2008

On Thursday, March 27th 2008, a DoS-vulnerability in lighttpd 1.4.19 and lower versions was reported on cve.mitre.org. It has been confirmed through lighttpd's bug-tracking system. As of this writing, there is no official bugfix-release of lighttpd, yet, but according to the bug-tracking system, the issue is closed and marked as fixed. This suggests that there might be a new release soon, especially as this has been discussed in several comments on the bug-tracking system.

The problem as described in the bug-tracker: "if a user killed his ssl connection, lighttpd would kill another ssl connection as it didn't clear the ssl error queue."