The Rackspace Fallout: Details Scarce After Weekend-Long Email Outage

The Rackspace outage, which put thousands of email users offline over the weekend due to a cybersecurity event, is a lesson for partners in disaster planning and customer support.

The yet-to-be-disclosed security incident took down Rackspace’s Microsoft Hosted Exchange environments. The cloud services provider early Monday morning announced that it had resorted email services “to thousands” of Microsoft 365 customers, but partners and customers continue to voice their concerns about the vendor’s response to the crisis.

StratusDial’s Bill McClain

“We’re looking at it they’re never going to be able to recover on that platform. We’ve moved on,” said Bill McClain, founder and president of MSP Networking Solutions and voice provider StratusDial.

Rackspace early on Friday announced that it was investigating connectivity and login issues in its Hosted Exchange environments. The company turned off the environment to further analyze the problem.

Rackspace continued to describe the problem as “connectivity issues” in multiple updates throughout the morning. A day later on Saturday, Rackspace concluded that a “security incident” had impacted the environment. It added that the Rackspace email product line and platform had not experienced any problems. Only Hosted Exchange accounts had suffered the incident, according to the company.

Rackspace encouraged customers to open new Microsoft 365 accounts and offered them free Microsoft Exchange Plan 1 licenses. It added on Saturday that it had dispatched 1,000 people to support customers. The company said employees made contact with customers either by phone or an alternative email address, although many customers complained on social media about long waits on call support lines.

Rackspace when contacted by Channel Futures pointed to the last update it made on its website.

“We continue to make progress on restoring email service to every affected customer,” the company wrote. “At this time, moving to Microsoft 365 is the best solution for customers who can now also implement temporary forwarding. To assist customers through options, Rackers are contacting every Hosted Exchange customer by phone. Customers will also be contacted via alternate email addresses. This outreach is being performed in addition to chat, phone, and ticketing.”

Customer Impact

Nevertheless, the outage drew complaints from customers on Twitter.

Tiffany Dowd, president and founder of the social media consulting company Luxe Social Media, took to Twitter to air her frustration.

“There has been a major outage affecting small businesses worldwide – including mine! No emails and I’m traveling on business for several weeks. Unacceptable,” she wrote.

Attorney Ronald Richards criticized how Rackspace communicated throughout the weekend.

Since Friday morning, millions of people have been denied access to the Exchange accounts. @Rackspace has basically recommended customers open up new accounts with other companies. They have kept people in the dark with very bad update messages. No one really knows. #rackspace🆘

— Ronald Richards (@RonaldRichards) December 4, 2022

Iman Jalali, managing partner at Bear Peak Capital, pointed to layoffs as a reason for the delayed restoration and subpar support.

36 hours ago the email hosting service @Rackspace went down

With it thousands of businesses lost access to their emails and it’s still down

They used to be one of the best hosting providers, but then they laid off a large portion of their staff… not a coincidence.

— Iman Jalali (@Stealx) December 3, 2022

Security Culpability?

Details remain to be seen about the actual security incident and any security posture from Rackspace that may have led to the problem. However, partners have pointed to systemic failings.

Lawrence Technology Systems’ Tom Lawrence

“Just from the standpoint of them not being able to get the systems back up in a reasonable amount of time and running shows there is some type of failure in their disaster recovery process,” said Tom Lawrence, who runs the Michigan-based MSP Lawrence Technology Services.

That said, Lawrence said he isn’t jumping to too many conclusions.

“I do try to avoid victim blaming because the reality of these attacks is that threat actors are doing these things for profit, and until there is a debrief we don’t know if they had an adequate level of protections or not. Was this a new sophisticated attack or were their servers left unpatched? We just don’t know yet,” he told Channel Futures.

Lawrence added that Microsoft’s slowness to respond with a method for patching issue compounded the problem.

Peter Radizeski, founder and president of the advisory firm Rad-Info, emphasized the importance of how a vendor responds to a disruption like this.

Rad-Info’s Peter Radizeski

“Outages happen. Period. How the vendor responds to that outage is the difference,” Radizeski told Channel Futures. “In my 23 years of experience, most outages are treated like a nuisance. They should be treated the way Tylenol reacted to poisoning in the 1980s. It is crisis management. It is reputation management. Being pro-active and forthcoming in communications is the key to maintaining trust. Everyone understands today that outages happen – every provider has had a major one at this point: AT&T, Rogers, Lumen, Verizon, Netflix, AWS, Rackspace. How the company reacts is what partners are looking at – especially for the coveted enterprise accounts. No one will risk moving a large account to a provider who fails during an outage. And the funny thing is, communicating during this time isn’t that difficult,” Radizeski told Channel Futures.

Partner Response

McClain said members of his team were working over the weekend to restore email access to customers. McClain said the first ticket from a customer appeared at 6:30 in the morning on Friday followed by a steady stream of tickets. Because Networking Solutions uses hybrid domains from Rackspace, it needed to migrate all of its Rackspace users to new accounts – even those not using Hosted Exchange.

McClain said his team has moved approximately 150 users and about 600 mailboxes and is nearing completion. They called Rackspace on Friday and didn’t get an answer. They called again on Saturday, also to no avail.

“I pretty much told my team, ‘We have to do whatever we have to and assume we’re not going to have any help whatsoever from Rackspace,’ and that was the case,” McClain told Channel Futures..

Part of the challenge was the uncertainty about..-