“I know how much you rely on our services.”
Facebook boss Mark Zuckerberg has apologised for last night’s lengthy downtime which hit the social network and all its connected services.
Oculus games and apps were left unavailable to access, while games which use Facebook as a login method (such as Pokémon Go) were also hit.
“We’re aware that some people are having trouble accessing our apps and products,” Oculus wrote on Twitter last night – a canned message shared by many Facebook services. “We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.”
Oculus – as well as Facebook, Instagram, Messenger and WhatsApp – remained unavailable for around six hours.
Reports from inside Facebook HQ painted a chaotic scene with employees unable to access internal security systems and email.
“Sorry for the disruption today,” Zuckerberg posted on Facebook, when his company’s many services began to come back online. “I know how much you rely on our services to stay connected with the people you care about.”
We?re aware that some people are having trouble accessing our apps and products. We?re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Oculus (@oculus) October 4, 2021
This content is hosted on an external platform, which will only display it if you accept targeting cookies. Please enable cookies to view.
So, what caused the issue? Despite rumours of some kind of massive DDOS attempt, all signs pointed to a routine update within Facebook accidentally spiralling out of control.
“Facebook (accidentally, we assume) sent an update to a deep-level routing protocol on the internet that said, basically, ‘hey we don’t have any servers any more xoxo’,” The Guardian’s Alex Hern explained on Twitter in a widely-shared thread. “Normally, this would be quite easy to fix… but Facebook runs everything through Facebook.
“So when its servers were booted off the internet, it also booted off the ability to send that follow-up message. And the ability to log-in to the system that would send the follow-up message.
“And the ability to use the smartcard door lock on the front door to the building that contains the servers that control the system that sends the follow-up message.
“And the messaging service you use to contact the head of physical security to tell them they need to high-tail it to the data centre out east with a physical key to override the smartcard door lock on the front door.”
Oops.
