Recently our email servers have come under sustained brute force attacks by script kiddies doing dictionary attacks. These go on for 24 hours a day from a variety of sources including pools of IP addresses that alternate probes from a common dictionary. These were flooding the maillog with authentication errors at a rate in excess of one every 10 seconds or so.
Iptables in the Linux network stack has the ability to look inside of a packet and match a string. We’ll use that feature to pick out authentication errors on the outbound side in order to block them on the inbound side.
I’m going to put this in the format of the CentOS/Redhat IPTABLES service that gets started by the init scripts. I’ll leave to the reader to translate that into a script.
# Firewall configuration written by system-config-securitylevel # Manual customization of this file is not recommended. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :RH-Firewall-1-INPUT - [0:0] # add the following # create the tables that we need :SMTP_Brute_Force_INPUT - [0:0] :SMTP_Check_Auth_OUTPUT - [0:0] :SMTP_Check_Auth_INPUT - [0:0] :SMTP_LOG_1 - [0,0] :SMTP_LOG_2 - [0,0] :SMTP_LOG_3 - [0,0] # add jumps for NEW connections to our filters on the INPUT chain for the SMTP and SUBMISSION ports -A INPUT -p tcp -m multiport --dports 25,587 -m state --state NEW -j SMTP_Brute_Force_INPUT -A INPUT -p tcp -m multiport --dports 25,587 -m state --state NEW -j SMTP_Check_Auth_INPUT # end of customization -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT # add the following # add jumps for ESTABLISHED connections to our filters on the INPUT chain for the SMTP and SUBMISSION ports -A OUTPUT -p tcp -m multiport --sports 25,587 -m state --state ESTABLISHED,RELATED -j SMTP_Check_Auth_OUTPUT # end of customization # the rest of the RH-Firewall-1-INPUT filter -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT # .... # .... -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited # Add the Brute force filter on the INPUT side # this is a netblock that belongs to us so RETURN. Add as many of these as you wish -A SMTP_Brute_Force_INPUT -s 123.123.123.0/24 -j RETURN # set an entry in the recent table -A SMTP_Brute_Force_INPUT -m recent --set --name SMTP --rsource # If we haven't had 6 connection attempts in the last minute then RETURN -A SMTP_Brute_Force_INPUT -m recent ! --rcheck --seconds 60 --hitcount 6 --name SMTP --rsource -j RETURN # log the event -A SMTP_Brute_Force_INPUT -j LOG --log-prefix "SMTP Brute Force Attempt: " # and REJECT the connection -A SMTP_Brute_Force_INPUT -j REJECT --reject-with icmp-port-unreachable # Add the authentication filter on the OUTPUT side # one of our netblocks so RETURN -A SMTP_Check_Auth_OUTPUT -d 123.123.123.0/24 -j RETURN # if the contents packet do NOT have the authentication error string then RETURN - customize for your mailserver -A SMTP_Check_Auth_OUTPUT -p tcp -m string --to 120 --algo kmp --string ! "535 5.7.0 authentication failed" -j RETURN # set an entry in the recent table -A SMTP_Check_Auth_OUTPUT -p tcp -m recent --name SMTP_AUTH_ERROR --set --rdest # GOTO (no RETURN) the logging tables # fun to watch in the syslog - adjust the frequency to suit. This is 3 strikes in 20 minutes -A SMTP_Check_Auth_OUTPUT -m recent --rcheck --name SMTP_AUTH_ERROR --seconds 1200 --hitcount 3 --rdest -g SMTP_LOG_3 -A SMTP_Check_Auth_OUTPUT -m recent --rcheck --name SMTP_AUTH_ERROR --seconds 1200 --hitcount 2 --rdest -g SMTP_LOG_2 -A SMTP_Check_Auth_OUTPUT -m recent --rcheck --name SMTP_AUTH_ERROR --seconds 1200 --hitcount 1 --rdest -g SMTP_LOG_1 # add the logging filters - these RETURN back to the OUTPUT table -A SMTP_LOG_1 -j LOG --log-prefix "SMTP_AUTH_FAIL: Strike 1: " -A SMTP_LOG_2 -j LOG --log-prefix "SMTP_AUTH_FAIL: Strike 2: " -A SMTP_LOG_3 -j LOG --log-prefix "SMTP_AUTH_FAIL: Strike 3: " # Add the target for the INPUT side # we are here because this is a new connection - if there hasn't been 3 hits in 20 minutes then RETURN - adjust to your needs -A SMTP_Check_Auth_INPUT -m recent ! --rcheck --name SMTP_AUTH_ERROR --seconds 1200 --hitcount 3 --rsource -j RETURN # log the event -A SMTP_Check_Auth_INPUT -j LOG --log-prefix "SMTP_AUTH_REJECTED: Go AWAY: " # tag it again -A SMTP_Check_Auth_INPUT -p tcp -m recent --name SMTP_AUTH_ERROR --set --rsource # and REJECT the connection -A SMTP_Check_Auth_INPUT -j REJECT --reject-with icmp-port-unreachable
A couple of notes about the authentication error message “535 5.7.0 authentication failed”. This will vary from server to server. You can find out what it is by doing a telnet connection to your mailserver and simulating a SMTP logon session.
[root@mail ~]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 mail.ourdomain.com ESMTP Sendmail 8.13.8/8.13.8; Tue, 26 Mar 2019 12:56:40 -0700 EHLO me.com 250-mail.ourdomain.com Hello localhost.localdomain [127.0.0.1], pleased to meet you 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-8BITMIME 250-SIZE 250-DSN 250-ETRN 250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN 250-STARTTLS 250-DELIVERBY 250 HELP AUTH LOGIN 334 VXNlcm5hbWU6 dXNlcm5hbWUuY29t 334 UGFzc3dvcmQ6 bXlwYXNzd29yZA== 535 5.7.0 authentication failed QUIT 221 2.0.0 mail.ourdomain.com closing connection Connection closed by foreign host.
The two strings following the 334 questions are the encrypted USER and PASS entries. The 535 is the authentication failure string needed in the iptables filter. Also note thet I spent a fair bit of head banging time because the search algorithm failed in the following.
-A SMTP_Check_Auth_OUTPUT -p tcp -m string –to 120 –algo kmp –string ! “535 5.7.0 authentication failed” -j RETURN
I was using
-A SMTP_Check_Auth_OUTPUT -p tcp -m string –to 120 –algo bm –string ! “535 5.7.0 authentication failed” -j RETURN
From the iptables man page.
–algo bm|kmp
Select the pattern matching strategy. (bm = Boyer-Moore, kmp = Knuth-Pratt-Morris)
The Boyer-Moore pattern matching simply did not work. Don’t know why.
Here is a sample from my /var/log/messages
Mar 26 13:02:50 mail saslauthd[3664]: do_auth: auth failure: [user=eblackford] [service=smtp] [realm=mydomain.com] [mech=pam] [reason=PAM auth error] Mar 26 13:02:50 mail kernel: SMTP_AUTH_FAIL: Strike 3: IN= OUT=eth0 SRC=123.123.123.201 DST=193.248.55.198 LEN=73 TOS=0x00 PREC=0x00 TTL=64 ID=31837 DF PROTO=TCP SPT=25 DPT=60937 WINDOW=46 RES=0x00 ACK PSH URGP=0 Mar 26 13:03:26 mail kernel: SMTP_AUTH_REJECTED: Go AWAY: IN=eth0 OUT= MAC=00:0c:29:10:b0:71:00:90:fb:31:9a:d5:08:00 SRC=93.157.63.7 DST=123.123.123.201 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=7624 DF PROTO=TCP SPT=5658 DPT=25 WINDOW=29200 RES=0x00 SYN URGP=0
And here is a tcpdump statement so that you can debug the authentication attempts. Another method of picking out the authentication error string
tcpdump -ni eth0 -A -s 120 -A 'tcp port 587 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' | less
13:19:28.990382 IP 123.123.123.201.smtp > 193.248.55.198.57597: P 427:460(33) ack 66 win 46 ...1....)..q..E..I+-@.@.....]...7.....J.....{P...$...535 5.7.0 authentication failed
The net result of this exercise is that two days later there are still a half dozen IP addresses in Russia, China, and the Netherlands that are still pounding away at my server. About once every 20 minutes each gets a chance at an authentication failure. I wish them luck.
What would be really fun would be to build an SMTP server in in Python or Perl that you could port forward to for the connections that are established culprits. You could give a series of 451 messages such as
451 Your mother wears army boots
451 May the fleas of a thousand camels infest your armpits
451 Go away – give up.
Or even better, reply to random authentication attempts with success! That would get their attention and confuse the hell out of them. And make the entire exercise pointless. How would they know what credentials were legitimate? They would either go away or launch a DOS attack on you. LoL
Have fun.