Web site attacked

You may have noticed that our website was down for a while last weekend. That’s because it was under attack. In fact, it has been under attack a lot longer than that. Only now it was bad enough to make the website inaccessible. Luckily, I was still able to log in with SSH. top confirmed the heavy load with load averages of above 40 and Apache using the most resources. Some investigation of Apache’s logs showed a lot of attempts to create a new account and to log in:

117.26.249.115 - - [05/Jan/2014:06:25:41 -0500] "GET /PmwikiVsMoin?action=newaccount HTTP/1.1" 200 3051 "http://gnewsense.org/PmwikiVsMoin?action=login" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
23.90.28.237 - - [05/Jan/2014:06:25:41 -0500] "GET /PmwikiVsMoin?action=login HTTP/1.0" 200 2915 "http://gnewsense.org/PmwikiVsMoin" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
192.227.240.168 - - [05/Jan/2014:06:25:41 -0500] "GET /Main/Deltah HTTP/1.0" 200 3690 "http://www.gnewsense.org/Main/Deltah?action=login" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
222.77.200.70 - - [05/Jan/2014:06:25:41 -0500] "GET /PmwikiVsMoin HTTP/1.1" 200 4253 "http://gnewsense.org/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
81.52.143.30 - - [05/Jan/2014:06:25:41 -0500] "GET /matkinson HTTP/1.1" 200 3063 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)" 
117.26.249.115 - - [05/Jan/2014:06:25:46 -0500] "GET /PmwikiVsMoin HTTP/1.1" 200 4253 "http://gnewsense.org/PmwikiVsMoin?action=login" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
23.90.28.237 - - [05/Jan/2014:06:25:46 -0500] "GET /PmwikiVsMoin?action=newaccount HTTP/1.0" 200 3068 "http://gnewsense.org/PmwikiVsMoin?action=login" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
192.227.240.168 - - [05/Jan/2014:06:25:46 -0500] "GET /Main/Deltah?action=login HTTP/1.0" 200 2919 "http://www.gnewsense.org/Main/Deltah" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
91.236.74.103 - - [05/Jan/2014:06:25:46 -0500] "GET /Main/No-Fuss_pc_games_Solutions_-_An_Update?action=login HTTP/1.1" 200 2958 "http://www.gnewsense.org/Main/No-Fuss_pc_games_Solutions_-_An_Update" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
222.77.200.70 - - [05/Jan/2014:06:25:46 -0500] "GET /PmwikiVsMoin?action=login HTTP/1.1" 200 2915 "http://gnewsense.org/PmwikiVsMoin" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
117.26.249.115 - - [05/Jan/2014:06:25:46 -0500] "GET /PmwikiVsMoin?action=login HTTP/1.1" 200 2915 "http://gnewsense.org/PmwikiVsMoin" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" 
91.236.74.103 - - [05/Jan/2014:06:25:47 -0500] "GET /Main/No-Fuss_pc_games_Solutions_-_An_Update?action=newaccount HTTP/1.1" 200 3110 "http://www.gnewsense.org/Main/No-Fuss_pc_games_Solutions_-_An_Update?action=login" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36"

I wrote a little awk script to count the unique IP addresses in the access log of the past 6 days:

{print $1; ipcount[$1]++; print ipcount[$1];} 
END {for (ip in ipcount) { 
        print ip, ipcount[ip] 
    }}

More than 50000 requests came from a single IP address, dozens of other IP addresses made between 1000 and 10000 requests. Also note the spammy page name in the last entry of the excerpt above. This smells an awful lot like a spammer’s botnet is trying to abuse our wiki for its nefarious purposes.

We’d already taken some measures in the form of textcha’s and a self-organizing access control group. This was enough to stop actual spam on the wiki. The access control group alone would have been enough to stop edits, but the textcha also needs to be filled in when you create a new account on the wiki. We left it in place as a barrier for the spammer to create new accounts. I doubt that the 114 new accounts that were created in the past 2 months are all legitimate, so that slowed it down at best. Unused accounts aren’t too bothersome, so we left it at that.

Now it seems that our spammer has grown tired of being slowed down and has picked up the pace. Apache and MoinMoin were working so hard to process all these new account and login requests that regular users could hardly get through. The spammer still didn’t manage to edit the wiki and the number of accounts didn’t explode, but we’d rather stop all this unwanted traffic at the front door. My predecessor, Karl Goetz, had already installed and configured Fail2ban in an earlier attempt to combat wiki spam.

Fail2ban is a tool that checks log files for suspicious behavior and blocks the offender for a while. I figured that a normal user would not need to create a new account more than twice within a few minutes. So I told Fail2ban to look out for these requests by putting an appropriate regular expression in filter file /etc/fail2ban/filter.d/moinmoin-login.local:

[Definition] 
failregex = ^<HOST> - -.*GET.*action=newaccount HTTP.*$ 
ignoreregex =

An accompanying section in /etc/fail2ban/jail.local tells it to look in Apache’s logs and email some details about the spammer:

[moinmoin-newaccount] 
enabled = true 
filter = moinmoin-newaccount 
logpath = /var/log/apache2/org.gnewsense.access.log 
findtime = 600 
maxretry = 2 
port = http,https 
action = %(action_mwl)s

A few minutes after restarting Fail2ban I could see from “iptables -n -L fail2ban-moinmoin-newaccount” that the firewall rules were trickling in. I made a similar filter and jail for “action=login”. Now the firewall is steadily blocking around 45 IP addresses for each of the filters and the load averages on the server are back to single digits before the decimal character. There should be no problem for regular users to create a new account or log in. Tor users could be affected, but that means that more people should run an exit node.

Author: Sam Geeraerts, gNewSense Project Leader.


CC0

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s