Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
To be fair to GitHub, outages are pretty short (~40 minutes), and they rarely impact critical parts of the service, nor do they affect one’s ability to work (git offline)
git push ending up with Internal Server error is similar to save file ending up with disk full error. yes, it can work after 40 minutes, and one can go to lunch and retry later, but here in Europe the troubles started at the end of a (normal) workday.
I'm not able to get code pushed, meaning I can't address PR feedback, meaning I can't get things approved, meaning code isn't getting merged to main and thus a ticket isn't getting closed. Sounds pretty critical to me.
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
[0] https://news.ycombinator.com/newsguidelines.html
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
C-x M-c M-unfuck-git
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
Maybe 30 minutes to get it all setup and then I moved on with my life.
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
I noticed because SSO was down. They don't even report on that as far as I can tell.
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
https://github.com/about