Case Study Date: 22nd of december 2015

Background

This client uses their server for around 150 web sites, mostly wordpress sites for their dealer network located worldwide. They have an in house IT person who takes care of the email and printers around the head office, a person based in Europe who takes care of their SEO and wordpress sites and a server person located in Puerto Rico who takes care of (or was) the web servers.

Their server person set up an Amazon EC2 server and installed cPanel and looking at the system after we came on the scene nothing else was done.

Basically, it was a vanilla cPanel install. Zero security footprint. Very Reckless.

Why we were contacted

The client was complaining of:

  • Server being blacklisted on RBL’s/URLBL’s
  • One of their sites being continually hacked
  • Server going down for a whole weekend (due to server running out of space)

Initial Observations

Here is what we found after logging ito the server:

  • SSH Port Open to the public
  • SSH Password Auth was being used
  • Able to login directly to WHM with root password
  • No firewall installed
  • All ports open
  • All none essential services open
  • 5,900,000 emails in the exim queue (yes you read it right: 5.9 MILLION spams)
  • No rootkit hunter or clamav or any scanning tools
  • Hacked WordPress sites

First Job: Lock Down the Server

When we lock a server down, we perform over 30 + tasks to ensure that the server is safe as it can possibly be in terms of root access, stability, monitoring and scanning without hindering the visitors to the sites.

This involves configuring firewalls, mod security, port blocking, scanning + more.

A complete list of those tasks can be found here.

Second Job: Clean up the mail queue

We have never actually seen a server so badly open for relaying and spamming. We have seen 100K emails on a server at most. This server had close to 6 million! After locking down exim we started running a script to delete all the spam emails. It took appx 48 hours to clear the queue and bring it back to normal. The irony is that all the wordpress sites are actually using Amazon SES to send emails so exim does not even really need to be enabled.

case-study-2-amazon-spam

Third Job: Clean up the hacked sites

We ran maldetect and CXS scanner from ConfigServer to identify any malicious files. There were around 70 infected files across the whole /home folder and this was surrounding two sites. One of the sites merely had some script injected into a gravity forms plugin and was basically sending out thousands of email a minute, the other was having random malicious files written to various folders and we think it was using forwarding to hack other wordpress sites on other remote server.

After running WPSCAN on one of the infected sites we determined that the hack was due to an outdated plugin.

NOTE: this site already had WordFence installed which did not detect the hacked files nor prevent the hack.

case-study-2-amazon-wpscan

NOTE: 3 days after we started on the job, the client decided to forward an email they received 20 days earlier from the Amazon abuse team saying that their server was being used to hack other web servers and their service was being throttled. We were amazed that their server guy did not take any action on this.

case-study-2-amazon-abuse

After locking the server down, we sent a request back to Amazon to confirm the server was no longer under threat and no throttling was taking place and they responded in the positive.

case-study-2-amazon-abuse-solved

Fourth Job: Reduce Unnecessary CPU Usage

Whilst analysing the error_log, we noticed an unusual amount of WordPress bruteforcing on the WordPress login pages. The wordpress sites existing on the server don’t use any custom directories so hackers and bots are able to easily find the WordPress login. The result being thousands of login attempts every minute. We installed a .htaccess in the /home directory with a main username and password. This stopped the attacks and reduced the server load immediately.

reduced-cpu-usage

Fifth Job: Reduce Backup Space and Disk Usage on Server

We ran a couple of backups of all sites in the /home folder and noticed the backup was around 100GB each time. After investigation, we discovered that there were multiple backups of each site sitting in their own public_html directory.

case-study-2-amazon-backups

After removing all static backups, the storage on the server in the /home directory dropped from 67% to 27%. A saving of 40%. This means that less storage is used and nightly backups will be 40% smaller which also means 40% less bandwidth being used.

case-study-2-amazon-disk

Conclusion

The client had been pulling their hair out managing their server “in-house” and had been given bad advice at each and every turn. They were relying on advice from people who had basic knowledge in production web servers and thus experienced serious problems which resulted in possible loss of business.

In this case we were able to shore up the server whilst keeping it online and optimise the speed of the server through tweaking the services and adding layers of security to prevent further issues.