Early this morning we discovered a customers out-of-date WordPress installation on one of our servers had been compromised. The compromised website was then used to send out large amount of junk email.
As soon as the problem was discovered, the website was disabled and all remaining junk email removed from any outbound queues.
Unfortunately, as a result, a few external DNS Blacklists (RBLs) have temporarily listed the affected server. We have requested removal from these RBLs and are waiting for the blocks to be removed. In the short-term, some customers may find some emails to certain providers are temporarily rejected until the blacklists are fully removed.
Update 18:50: An engineer has attended the site and the problem has now been resolved.
At approximately 17:15 this evening, a problem was detected on our legacy Zebra CMS hosting platform.
An engineer has been looking into resolving the issue, but efforts to resolve the problem remotely have not been successful. An engineer is now heading to the data centre to investigate further.
We will update this post with further information as soon as possible.
Update 24th May: All customer data has been migrated to an alternative server. The old server has been removed from service pending investigations into the issues encountered. All services continue to operate normally.
Over the past few days we’ve been receiving reports of issues with customers sites & email located on one of our hosting servers.
Whilst under investigation at approximately 09:00 BST yesterday the performance of this server drastically reduced. This meant that the vast majority of customers hosted on that server were no longer able to send or receive email, causing a large backlog of undelivered emails. This compounded the existing performance issues the server was suffering, causing performance to drop yet further.
We have now identified an issue with the server’s disk storage system. To help alleviate the issue we have migrated all affected customer mailboxes over to spare capacity on another server. Due to the large amount of email involved this took a number of hours to complete. Email services started coming back online at approx. 16:00 BST yesterday and were fully operational again at 17:00 BST. Any emails which were delayed yesterday should be delivered soon if they have not already been delivered.
Whilst we have alleviated the immediate problems, we now need to fix the main issue permanently. Our engineers and vendors are in discussion as to the best resolution and we will update you once we have further information.
If your services are still not functional please call us on 0191 2612252 and we can investigate on a case by case basis, but please be aware that we are experiencing extremely high call volumes.