ConnectWise Cloud: Dark Until Oct. 28?
A single growing pain or real problems? ConnectWise apparently is working to restore its hosted PSA service. Problems seem to have started mid-day October 26 and there are indications full service won’t be restored until Oct. 28. Here are some of the details.
Updated 10:49 a.m. eastern, Oct. 28: ConnectWise CEO Arnie Bellini informs me that service was restored to the 198 affected partners at 10:30 a.m. eastern on Oct. 28.
Updated 10:54 p.m. eastern, Oct. 27: Please see ConnectWise CEO Arnie Bellini’s comments below, in which he clarifies the scope and status of the outage.
My original Oct. 27 blog entry resumes here:
I’ve got a note and voicemail into ConnectWise CEO Arnie Bellini and COO Santo Cannone. As soon as I get a reply I will share some more details.
In the meantime you can find some of the blow-by-blow out on Twitter, where hosted ConnectWise customers like Mike Joseph started describing problems mid-day Oct. 26. The chatter soon spilled over to Larry Walsh’s blog, where he reported:
ConnectWise says about 100 of its Cloud customers will remain without service until Wednesday morning following a major hardware failure in its Web-based service infrastructure on Tuesday.
Bad Timing
The outage is a case of awful timing for ConnectWise, which is hosting a major partner conference in Orlando Nov. 4-6.
In recent months, ConnectWise has somewhat altered its messaging and pricing to promote Hosted and SaaS expertise — taking an aggressive stance against rival Autotask.
While Autotask has long promoted itself as a pure SaaS and cloud option, the vast majority of ConnectWise deployments involve on-premise engagements.
No doubt, on-premise software companies and even Internet-centric giants have hit bumps from time to time when introducing expanded cloud and SaaS services. Microsoft suffered a storage cloud black eye earlier this month. And Amazon.com’s Simple Storage Service (S3) has suffered high-profile outages.
In ConnectWise’s case the affected user base apparently is limited to a small percentage of users (since most ConnectWise customers are on-premise) but you can bet the SaaS folks at Autotask are watching the competitive situation closely.
What caused the ConnectWise outage? I hesitate to speculate, and will wait to hear an official explanation from ConnectWise. But the timing certainly hurts ahead of the ConnectWise Partner Summit.
Follow MSPmentor via RSS; Facebook; Identi.ca; and Twitter. And sign up for our Enewsletter; Webcasts and Resource Center. Plus, check out more MSP voices at www.MSPtweet.com.
Hi Folks. The note below apparently was sent from ConnectWise to Hosted users explaining the outage earlier today:
“I am writing to bring you up to date on the interruption in service on the ConnectWise Cloud (Hosted) service. We experienced a major hardware failure earlier today and have attempted to repair the system, with no success.
We are now in the process of restoring the system to a new configuration using a backup taken just before the outage. This restore will take several hours, so you will be unable to access ConnectWise until the start of your business day tomorrow.
We know that ConnectWise is critical to your business. We apologize for the interruption in service and the impact it is having on your business. We will be issuing a credit for the lost service on your next invoice. We will also be doing a full review of the situation and assuring that we are able to recover from future outages with minimal impact on you. We will update you when that review is completed.”
Dear Arnie,
I can’t say I’m happy about the outtage but you’ve always gone the extra mile for partners like me. You’ve earned my loyalty. I look forward to hearing what caused the outtage and the steps you are taking to avoid a repeat of the situation.
JT
Hi Joe,
I couldn’t help but notice the irony here. I think you tweeted earlier today that cloud usage has increased more than 300 percent since January 2009. Thanks for your continued reality checks here and on The VAR Guy. (Hey, are YOU the VAR Guy?)
Frankie
JT@2: Can you share any more details about who you are and how long you’ve been running Hosted ConnectWise?
Frankie@3: As the old Airplane II line goes, “Irony can be pretty ironic sometimes.” As for The VAR Guy, I’ve never met the guy. I just co-fund his web site. Thanks for reading.
-jp
We have a portion of our cloud down. Any outage is unacceptable, we know it and are working thru the night to correct it. We are truly sorry to the 198 partners that are affected.
Our United Kingdom cloud is 100% operational and our Exchange Email cloud is 100% operational. Our United States cloud for ConnectWise PSA has a partial outage that is effecting 198 partners. Again, any outage, even one partner, is unacceptable. We have a large investment in virtualizing our cloud. These 198 partners are the only remaining partners on our legacy environment and are not in the cloud, they are on redundant set of servers that have experienced a major outage. Leasson learned, there is never too much redundancy! We are working through the night and expect to have service restored by Wed morning, exact times I don’t yet know.
Once service is restored we will rapidly complete the migration work to get these 198 partners over to the virtualized cloud. Wish we had been ahead of schedule on that work so our partners would not have experienced this outage. We will be sending the 198 partners a full credit for a month. Not that it will make up for the inconvenience but we want them to know that we are sorry. Sorry guys, we have learned from this and will make the necessary adjustments.
Arnie Bellini
CEO
ConnectWise
Arnie: Thank you for clarifying the situation. Please keep us posted.
-jp
Wow. I wish Microsoft had that kind of personal response.
Arnie thank you for helping me to win a bet with my business partner. I knew you’d post a comment before midnight Tuesday explaining the situation, apologizing and offering some sort of credit/refund to affected partners.
“It’s not what happens to you, but how you react to it that matters.”
Go IT Nation.
Roger*
*Not my real name because I don’t want sales inquiries from software vendors. I want software partners!
Stephen: Ditto
Still down. This is causing us serious revenue loss. If I didnt love ConnectWise, we might consider another vendor. We need you CW! Hurry Up.
Apex IT Group
We are still down, it is morning and we have no emails from support about the resolution…
It’s pretty scary to start thinking about your technological deck of cards and what happens if the foundation falls. Like most MSPs, if we lost our PSA we’d be a little lost in our minds and pocketbooks. It must be frustrating to those customers of Connectwise that are affected because it is out of their realm of control to fix (or prevent for that matter).
I think every good MSP needs a contingency plan if their technologies fail. For us? We keep around several stacks of old carbon paper service order forms!
My heart goes out to the 198 companies that have been unable to carry out their business operations due to this outage. At Tigerpaw, we understand the importance of the reliability of your business automation solution, which is one reason we still push on-premise for such a mission critical application.
I wish the 198 companies a speedy recovery.
James Foxall
Senior Vice President
Tigerpaw Software
I just received a note from ConnectWise CEO Arnie Bellini stating that the partial outage is fixed as of 10:30am eastern; ConnectWise says it has notified the 198 affected partners.
You cannot expect innovation without setbacks and mistakes. It takes a tremendous amount of risk to innovate, and without innovation we don’t have vendors who lead in this Channel. It certainly does tell us a lot about a vendor when that vendor makes a mistake, makes a humble acknowledgement of the mistake, and then does what is necessary to prevent the issue from happening again.
I suppose this is an opportunity for CW’s competitor’s to make jabs, but none of the Cloud solutions are infallible and as our most creative vendors continue to innovate as they migrate to the Cloud, I would think we’ll see more glitches and outages and service interruptions. For appropriate solutions, the Cloud is here and to not have a Cloud strategy is akin to putting your head in the sand.
In my opinion, it’s a good time as the Channel migrates solutions to the Cloud for resellers/msp’s/solution providers, etc. to ask ourselves what we’re doing to ensure our operations continue regardless of whether our Cloud Services fail. This, I would think, is the bigger question. We need to have good Contingency Plans in place for ourselves and our end users.
Amy Luby
CEO, MSPSN
aluby at mspsn dot com
http://www.mspsn.com
The partial outage is over. Service was restored to the 198 partners effected as of 10:28am this morning. We realize this is not acceptable and very frustrating and we are very sorry. The partners effected are on our legacy hosted environment, which is redundant, but had a major failure. We are going to aggressively move those partners over to our Virtualized Cloud to help assure that this won’t happen again. Again we are very sorry and will be providing all 198 partners a credit for a full month. That doesn’t make up for the inconvenience, but hopefully it says “we are sorry”. Thanks for your patience. We intend to learn and get better from this.
Just something else to think about when moving from on premise to hosted:
I just moved from on-premise CW to hosted CW. I have to say that everything went pretty smooth during the transition. I would rate the migration a 8-9 out of 10. Where I was disappointed was that during the sales process I was given an agreement that stated I would incur NO “One-Time Charges” about a month or two after the migration, I received an invoice for close to $1100 for what was labeled on the invoice “REINSTALL – Going from Onsite to Hosted”. When I complained about the invoice, they said they were willing to come down to close to $400. I stated that in black and white on the agreement I signed that it stated “no hidden fees”. And that the $1100 was nowhere to be found on my revised agreement. I would think that since it was not on the agreement (and unseen) this would actually be THE DEFINITION of a hidden fee! Their response was “The re-install was a billable instance at $175/hr. The one-time fees were waived, since you are an existing partner. The one-time fees cover accounting integration, implementation, etc, which you already have. There was only one install covered for your original agreement, and any re-installs are billable.”
But again, I was not told about any re-install fee.
I say all that to those thinking of moving from on-premise to hosted make sure you ASK ABOUT THE REINSTALL FEE! I am a one man shop, and $1100 dollars is a LOT of money for me to pony up without notice.
The fee to move from on-premise to hosted was clearly detailed to me. I was given the option of either $175/hr for the actual number of hours it would take, or a fixed fee of $475 (or something similar). This was back at the end of August. In the end I believe it took slightly less than two hours.
Thanks Arnie! You are great at what you do! Technology breaks sometimes! Please get us in the cloud.
George Mach
I guess the silver lining in this cloud is that as we expand the cloud to Australia and other continents we are becoming battle tested. It does help having the roots of an IT Solution provider. The lessons learned from this outage have made us more aggressive about virtualization and massive redundancy. We have allocated an additional $500k to make our Cloud bullet proof.
Arnie Bellini
CEO
ConnectWise
Great News! Keep doing great work! We stand behind you!
George