多个GitHub服务出现故障

2026年9月13日，UTC时间08:43至10:44之间，GitHub经历了大约28项服务的可用性下降，包括Issues、Pull Requests、Actions、Codespaces、Pages、Notifications、Code Scanning、Git LFS和新账户注册。在高峰时段，创建GitHub App安装访问令牌的请求中有8.8%失败。Actions工作流的令牌发放也受到影响，在事件时间段内影响了大约4%的工作流。通过Web界面创建问题的尝试中约有96%失败，注册失败率超过90%。<br /> <br />原因是内部数据清理作业于UTC时间07:33开始向共享数据库集群写入数据。该集群存储了几乎所有经过身份验证的请求都会读取的权限数据。用于调节该后台作业速度的保护机制只监控一个健康信号——数据库副本的滞后程度——而该信号全程保持低位。它没有考虑主库本身不断累积的负载，因此作业持续写入，而主库则悄然逼近其极限。<br /><br />当主库耗尽可用连接时，需要它的请求无法完成。首先，这些数据库调用没有快速超时机制，因此请求处理程序会等待停滞的数据库，而不是快速失败，共享的请求处理能力随之退化为全站错误。其次，令牌创建周围的重试循环不断重新发送已经失败的写入，使数据库持续处于饱和状态，无法恢复。<br /><br />监控在UTC时间08:50宣布进入事件状态，但由于影响范围广泛以及令牌创建带来的放大效应，识别负载来源花费了一些时间。第一响应者通过削减内部负载并暂停该作业来缓解问题，所有服务在UTC时间10:44恢复。<br /><br />为防止再次发生，我们默认对后台作业针对共享的、面向客户的数据库进行速率限制，并在主服务器负载而非仅复制延迟上添加自动暂停和告警。我们还将正在运行的后台工作直接展示在数据库健康信号旁边，以便响应者无需离开这些仪表板即可查看并暂停它，限制令牌发放路径中的重试次数，并添加请求级超时，使一个不健康的数据库无法消耗共享的Web服务器容量。此外，我们正在拆分该数据库集群以消除单点故障。我们将在未来两周内将各种服务特定数据（包括授权数据）移出该共享集群。

原文On September 13, 2026, between 08:43 and 10:44 UTC, GitHub experienced degraded availability across approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup. At peak, 8.8% of requests to create GitHub App installation access tokens failed. Token issuance for Actions workflows was also affected, impacting approximately 4% of workflows during the incident time frame. Creating issues through the web interface failed for about 96% of attempts, and signup failures were above 90%. <br /> <br />The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal — how far the database replicas were lagging — and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit. <br /><br />When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturated rather than letting it recover. <br /><br />Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC. <br /><br />To prevent recurrence, we are rate-limiting background jobs against shared, customer-serving databases by default, and adding automatic pausing and paging on primary-server load rather than replication lag alone. We are also surfacing running background work directly alongside database health signals so responders can see and pause it without leaving those dashboards, bounding retries in the token-issuing path, and adding request-level timeouts so one unhealthy database cannot consume shared web server capacity. In addition, we are breaking apart this database cluster to remove the single point of failure. We will be moving various service-specific data, including the authorization data, out of this shared cluster in the next two weeks.

Pull Requests 正经历性能下降。我们正在继续调查。

原文Pull Requests is experiencing degraded performance. We are continuing to investigate.

我们已通过内部减载降低了该集群的负载，并看到了恢复的迹象，但仍在继续监控。

原文We have reduced load on this cluster with internal load-shedding and are seeing signs of recovery but continue to monitor

我们正在观察到collab上数据库复制延迟增加，这导致授权端点的错误率上升，并进而引起整个系统错误率增加——我们正在调查中

原文We're seeing increased database replication delays on collab which is causing increased error rates in authorization endpoints and follow-on increased error rates across the system - we are investigating

Actions 正在经历性能降级。我们正在继续调查。

原文Actions is experiencing degraded performance. We are continuing to investigate.

我们正在调查有关API请求、问题、页面和拉取请求可用性下降的报告

原文We are investigating reports of degraded availability for API Requests, Issues, Pages and Pull Requests