CenturyLink’s Post-Holiday Outage: Timeline and Impact
What remains to be seen is whether CenturyLink will face any repercussions for the outage.
CenturyLink customers received an unwelcome post-Christmas surprise when their service went down Dec. 27 and stayed down through Dec. 28.
The lengthy outage resulted in everything from minor inconvenience for online shoppers, to restaurants and other businesses having to operate without internet, and spotty 911 service that created anxiety for those in need of help. Even Verizon was impacted. However, the outage reportedly didn’t result in any life-threatening incidents.
Still, federal and state officials want answers. The Federal Communications Commission (FCC) has launched an investigation and various states are looking into how their residents were impacted.
What remains to be seen is whether CenturyLink will face any repercussions for the outage.
Scroll through our timeline below to see how it all unfolded, as well as the aftermath that remains unsettled.
Post-Christmas Surprise
Many CenturyLink customers across the United States found themselves without internet on Dec. 27 amid a lengthy outage that stretched from New York to California.
CenturyLink tweeted the following that morning: "Our network is experiencing a disruption affecting customer services. We understand some customers are currently unable to generate tickets through the CenturyLink help portal. We know how important these services are, we are working to restore services as quickly as possible."
The outage began early in the day and continued into the evening. CenturyLink's network was "still experiencing a disruption," but the telecommunications company said it was working to restore services.
Many CenturyLink customers across the United States found themselves without internet on Dec. 27 amid a lengthy outage that stretched from New York to California.
CenturyLink tweeted the following that morning: "Our network is experiencing a disruption affecting customer services. We understand some customers are currently unable to generate tickets through the CenturyLink help portal. We know how important these services are, we are working to restore services as quickly as possible."
The outage began early in the day and continued into the evening. CenturyLink's network was "still experiencing a disruption," but the telecommunications company said it was working to restore services.
Verizon Customers Impacted
In addition to disrupting 911 services, the CenturyLink outage also caused outages of Verizon network services in at least two states: New Mexico and Montana. The company uses CenturyLink’s fiber strands to provide services locally.
“We ride our calls along with their network,” Verizon spokesman Steve Van Dinter told the Wyoming Tribune Eagle in a Dec. 28 report. “We’re still trying to get an estimate of when it will be back up online. Since it’s not our core network, we can’t be the ones to make that call.”
In addition to disrupting 911 services, the CenturyLink outage also caused outages of Verizon network services in at least two states: New Mexico and Montana. The company uses CenturyLink’s fiber strands to provide services locally.
“We ride our calls along with their network,” Verizon spokesman Steve Van Dinter told the Wyoming Tribune Eagle in a Dec. 28 report. “We’re still trying to get an estimate of when it will be back up online. Since it’s not our core network, we can’t be the ones to make that call.”
Ground Zero: Denver
The Denver Post reported that the Mile High City was "ground zero" for the outage.
For about 30 hours, from the early morning hours of Dec. 27 until late on Dec. 28, "chaos reigned" on CenturyLink’s system, it said. Western states that depend most heavily on the company’s fiber-optic system were hardest hit, but reports of outages and slower speeds came in from Alaska to Florida, according to downdetector.com.
The Denver Post reported that the Mile High City was "ground zero" for the outage.
For about 30 hours, from the early morning hours of Dec. 27 until late on Dec. 28, "chaos reigned" on CenturyLink’s system, it said. Western states that depend most heavily on the company’s fiber-optic system were hardest hit, but reports of outages and slower speeds came in from Alaska to Florida, according to downdetector.com.
Hospitals Hit
Banner Health's facilities and entities in multiple states were affected by the CenturyLink and Verizon outage that left staff at one Colorado hospital with irregular phone service and no access to patient records, according to the Greeley Tribune.
Greeley, Colorado,-based North Colorado Medical Center, owned by Banner, experienced outages for both phones and computers starting Dec. 27. Staff could not access patient records, and IT staff ensured acute care computers were working in hospital rooms.
Banner Health's facilities and entities in multiple states were affected by the CenturyLink and Verizon outage that left staff at one Colorado hospital with irregular phone service and no access to patient records, according to the Greeley Tribune.
Greeley, Colorado,-based North Colorado Medical Center, owned by Banner, experienced outages for both phones and computers starting Dec. 27. Staff could not access patient records, and IT staff ensured acute care computers were working in hospital rooms.
Back Online
Restoration of services began Dec. 27 and network traffic had normalized as of Dec. 29, said CenturyLink spokeswoman Linda Johnson.
“The outage was caused by a faulty network management card from a third-party equipment vendor that caused invalid traffic replication,” she said. “Steps are being taken to help prevent the issue from reoccurring.”
Restoration of services began Dec. 27 and network traffic had normalized as of Dec. 29, said CenturyLink spokeswoman Linda Johnson.
“The outage was caused by a faulty network management card from a third-party equipment vendor that caused invalid traffic replication,” she said. “Steps are being taken to help prevent the issue from reoccurring.”
FCC Launches Investigation
Federal Communications Commission (FCC) chairman Ajit Pai announced the agency is investigating the outage, but there's been no update since the FCC shut down most of its operations due to the Trump administration’s government shutdown.
“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” he said. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling. I’ve directed the (FCC’s) Public Safety and Homeland Security bureau to immediately launch an investigation into the cause and impact of this outage. This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”
Federal Communications Commission (FCC) chairman Ajit Pai announced the agency is investigating the outage, but there's been no update since the FCC shut down most of its operations due to the Trump administration’s government shutdown.
“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” he said. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling. I’ve directed the (FCC’s) Public Safety and Homeland Security bureau to immediately launch an investigation into the cause and impact of this outage. This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”
Washington AG Wants Answers
Washington state Attorney General Bob Ferguson wants to hear from residents who were impacted by the outage.
“For the second time, CenturyLink has fallen short of its obligation to provide reliable 911 services for Washingtonians,” he said. “If you called for help during this outage, only to be met with a busy signal, please share your story with my office. We want to know exactly how CenturyLink’s failure impacted the people of our state.”
The state’s Utilities and Transportation Commission fined CenturyLink nearly $2.9 million for a six-hour 911 outage in 2014, Ferguson said.
Washington state Attorney General Bob Ferguson wants to hear from residents who were impacted by the outage.
“For the second time, CenturyLink has fallen short of its obligation to provide reliable 911 services for Washingtonians,” he said. “If you called for help during this outage, only to be met with a busy signal, please share your story with my office. We want to know exactly how CenturyLink’s failure impacted the people of our state.”
The state’s Utilities and Transportation Commission fined CenturyLink nearly $2.9 million for a six-hour 911 outage in 2014, Ferguson said.
CenturyLink CEO Lays Blame
During last week’s 2019 Citi Global TMT West Conference in Las Vegas, CenturyLink president and CEO Jeff Storey blamed the outage on an equipment vendor that is a U.S.-based company and part of CenturyLink’s network, reiterating what a company spokesman had said on Dec. 29 (slide 5).
“The source of the outage was a particular equipment vendor and a malfunction with one of those (network management) cards,” he said. “It created an inability for the system to continue to process capacity and it blocked our ability to control those nodes. And so we had to physically go out and shut things down, and restart them on that transport layer. Meanwhile, we’re moving circuits off to other capacities. It was an equipment failure that had a more dramatic impact than we would have wanted it to have.”
During last week’s 2019 Citi Global TMT West Conference in Las Vegas, CenturyLink president and CEO Jeff Storey blamed the outage on an equipment vendor that is a U.S.-based company and part of CenturyLink’s network, reiterating what a company spokesman had said on Dec. 29 (slide 5).
“The source of the outage was a particular equipment vendor and a malfunction with one of those (network management) cards,” he said. “It created an inability for the system to continue to process capacity and it blocked our ability to control those nodes. And so we had to physically go out and shut things down, and restart them on that transport layer. Meanwhile, we’re moving circuits off to other capacities. It was an equipment failure that had a more dramatic impact than we would have wanted it to have.”
Readers Respond
Our coverage of the outage and aftermath prompted numerous comments from readers, such as that from Dawn Bozeman: "It is a shame that so many customers were affected by the outage." she wrote.
"We need to always remember that outages can be caused by many factors," she said. "Lost connections by cable cuts or other means are different from electronic component failures. Today, we see SD-WAN as an opportunity to re-route communications in the event of a circuit being degraded or lost. This is a great way to ensure connectivity to cloud-based applications, as you can have multiple routes in an SD-WAN design. However, it is imperative that the application being connected to also has a type or redundancy, or at least monitoring system to allow a quick response should a component fail."
Geri Wheat called the outage "unacceptable" and said it resulted in her modem going out, "and now CenturyLink wants me to buy another one for ($100)."
"The girl on the line said it was coincidence that this happened and basically 'oh well,' it’s Jan. 11 (and) I still have no service and no modem because I have basically been put on hold, transferred multiple times and everyone has an excuse and has not helped," he said.
Our coverage of the outage and aftermath prompted numerous comments from readers, such as that from Dawn Bozeman: "It is a shame that so many customers were affected by the outage." she wrote.
"We need to always remember that outages can be caused by many factors," she said. "Lost connections by cable cuts or other means are different from electronic component failures. Today, we see SD-WAN as an opportunity to re-route communications in the event of a circuit being degraded or lost. This is a great way to ensure connectivity to cloud-based applications, as you can have multiple routes in an SD-WAN design. However, it is imperative that the application being connected to also has a type or redundancy, or at least monitoring system to allow a quick response should a component fail."
Geri Wheat called the outage "unacceptable" and said it resulted in her modem going out, "and now CenturyLink wants me to buy another one for ($100)."
"The girl on the line said it was coincidence that this happened and basically 'oh well,' it’s Jan. 11 (and) I still have no service and no modem because I have basically been put on hold, transferred multiple times and everyone has an excuse and has not helped," he said.
