CNN
—
extreme Heat in California It left Twitter without one of its key data centers, and a company executive warned in an internal memo obtained by CNN that another outage elsewhere could cause a blackout for some users of the service.
twitter
(TWTR), like all major social media platforms, relies on data centers, which are huge warehouses full of computers, including servers and storage systems. Temperature control in these centers is essential to prevent computers from overheating and malfunctioning. To save on cooling costs, some technology companies are increasingly looking to locate their data centers in colder climates; Google, for example opened the data center In 2011 in Finland, Meta has had a center in northern Sweden since 2013.
“On September 5, Twitter lost its Sacramento (SMF) data center due to extreme weather. An unprecedented event resulted in a complete shutdown of the physical SMF hardware,” Carrie Fernandez, the company’s vice president of engineering, said in an internal message to Twitter engineers on Friday.
Large technology companies typically have multiple data centers, in part to ensure that their services remain online if one center fails; this is known as redundancy.
According to Fernandez’s Friday memo, Twitter is “in an emergency situation” as a result of the outage in Sacramento. He explained that Twitter’s data centers in Atlanta and Portland are still operating, but warned: “If we lose one of the remaining data centers, we will not be able to serve traffic to all Twitter users.”
The memo prohibits non-critical updates to the Twitter product until the company can fully restore its Sacramento data center services. “All production changes, including deployments and releases to mobile platforms, are blocked except for changes required to address service continuity or other urgent operational needs,” Fernandez said.
The restrictions highlight the apparent fragility of some of Twitter’s fundamental systems, an issue that former Twitter security chief Peiter “Mudge” Zatko raised in July in a statement to lawmakers and government agencies.
Zatko, whose disclosure was first reported by CNN and The Washington Post, warned that Twitter had “data center inefficiencies” that increased the risk of a brief service outage or even Twitter being offline forever.
“Even a temporary but overlapping outage of a small number of data centers will result in service [Twitter] being offline for weeks, months, or permanently,” according to Zatko’s whistleblower statement. (Twitter has criticized Zatko and widely defended itself against the allegations, saying the statement was a “false narrative” about the company.)
News of the data center outage comes a day before Zatko testifies before the Senate Judiciary Committee.
Twitter has not disclosed the number or locations of its data centers, but Zatko’s statement cited a public news report that identified a Twitter data center in Sacramento and Atlanta. in 2020 Amazon announced Twitter chose its cloud computing platform, Amazon Web Services, to serve some tweets from Amazon’s data centers.
Commenting on the Sacramento outage, a Twitter spokesperson told CNN: “At this time, there have been no disruptions affecting people’s ability to access and use Twitter. Our teams are equipped with the tools and resources to push updates and will continue to work to ensure a seamless Twitter experience.”
Retired Brigadier General Greg Touhill, who served as the US government’s chief information security officer in 2016 and 2017, said data centers “need reliable water, power, humidity control and refrigeration to survive.”
“You want redundancy, not redundancy, to increase your cyber resiliency in the event of a natural disaster [or other event] it can take down a single piece of equipment or a data center,” Touhill, who heads the CERT division at Carnegie Mellon University’s Software Engineering Institute, told CNN.
