Looks like Twitter is having their first very serious SEV since the acquisition! Mass user logouts, delays loading content. My guess is that this is very related to the recent shutdown of the Sacramento DC, although I’m still not clear on how far along that process is.
Feel free to chime in below with speculation, or just tweet in the comments until they get it fixed up.
(Also! Still hard at work on the Twitter’s Orgs series - got so much material together that I’m gonna have to go org-by-org. More on that soon!)
At fb/ig we had a system of... knobs that we could twiddle to turn down/off functionality as things got hot on the site - "online now" indicators for chat were always first to go (they did drive a lot more messaging activity than you'd expect, but were quite expensive to compute constantly).
I'm sure turning off ads was near the very bottom of that list (although maybe the matching of ads to users was higher up in the list - I know good matching made a lot of money for fb, but I also know that it was *very* computationally expensive)
Yeah. There was that big spike in errors during the opening ceremony, and some latency for images/vids during the final, but otherwise things have been just buggy.
I think somebody described what was going to happen to twitter as being like when you drive a car without any fuel. you'll be surprised that actually it will go for a bit, and think wow maybe it won't suddenly stop after all... and then it does.
Things that seem to be broken:
* Login for many people
* Users are force logged out
* Timeline sometimes failing to load
* I'm noticing very high latency loading notifications/comments, haven't confirmed with others
* _All_ ads seem to be gone for everyone (maybe a compute-saving measure?)
Oh wow, if you're at the point where you're killing your revenue to stay afloat, you must be desperate.
At fb/ig we had a system of... knobs that we could twiddle to turn down/off functionality as things got hot on the site - "online now" indicators for chat were always first to go (they did drive a lot more messaging activity than you'd expect, but were quite expensive to compute constantly).
I'm sure turning off ads was near the very bottom of that list (although maybe the matching of ads to users was higher up in the list - I know good matching made a lot of money for fb, but I also know that it was *very* computationally expensive)
Also! If y'all haven't seen them yet, check out the health charts by https://twitter.com/iFred - great nostalgia for any former SREs in the building:
https://deadbird.singlepane.io
I'm just surprised it's taken this long. I was sure it was going to be shortly after the world cup.
Yeah. There was that big spike in errors during the opening ceremony, and some latency for images/vids during the final, but otherwise things have been just buggy.
Slowly... and then suddenly.
I think somebody described what was going to happen to twitter as being like when you drive a car without any fuel. you'll be surprised that actually it will go for a bit, and think wow maybe it won't suddenly stop after all... and then it does.