‹ 返回事件歷史

Incident with several GitHub Services

已復原嚴重故障GitHub
2026年9月13日星期日 17:16 至 2026年9月13日星期日 18:44(1 小時 28 分鐘)
受影響元件
API Requests
Issues
Pull Requests
Actions
Pages
更新記錄
已復原2026年9月13日星期日 18:44

On September 13, 2026, between 08:43 and 10:44 UTC, GitHub experienced degraded availability across approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup. At peak, 8.8% of requests to create GitHub App installation access tokens failed. Token issuance for Actions workflows was also affected, impacting approximately 4% of workflows during the incident time frame. Creating issues through the web interface failed for about 96% of attempts, and signup failures were above 90%. <br /> <br />The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal — how far the database replicas were lagging — and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit. <br /><br />When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturated rather than letting it recover. <br /><br />Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC. <br /><br />To prevent recurrence, we are rate-limiting background jobs against shared, customer-serving databases by default, and adding automatic pausing and paging on primary-server load rather than replication lag alone. We are also surfacing running background work directly alongside database health signals so responders can see and pause it without leaving those dashboards, bounding retries in the token-issuing path, and adding request-level timeouts so one unhealthy database cannot consume shared web server capacity. In addition, we are breaking apart this database cluster to remove the single point of failure. We will be moving various service-specific data, including the authorization data, out of this shared cluster in the next two weeks.

排查中2026年9月13日星期日 18:28

Pull Requests is experiencing degraded performance. We are continuing to investigate.

排查中2026年9月13日星期日 18:26

We have reduced load on this cluster with internal load-shedding and are seeing signs of recovery but continue to monitor

排查中2026年9月13日星期日 17:36

We're seeing increased database replication delays on collab which is causing increased error rates in authorization endpoints and follow-on increased error rates across the system - we are investigating

排查中2026年9月13日星期日 17:25

Actions is experiencing degraded performance. We are continuing to investigate.

排查中2026年9月13日星期日 17:16

We are investigating reports of degraded availability for API Requests, Issues, Pages and Pull Requests

事件資料來自 GitHub 官方狀態頁:查看官方原文

關於這些資料

GitHub 的事件資料來自哪裡?

資料來自GitHub 官方狀態頁,每 5 分鐘同步,保留最近 90 天。詳情頁提供官方記錄連結。

如何解讀元件狀態列?

每格代表一天。綠色表示無異常記錄,黃色表示部分降級,紅色表示較大故障,灰色表示無資料。可用率統計涵蓋最近 90 天。

為什麼有些服務查不到歷史?

只收錄提供公開官方狀態頁的服務。

持續時間如何計算?

按官方標記的開始至復原時間計算。進行中的事件計算至目前時間。跨天事件會顯示在每個受影響的日期。

完整的服務清單與即時狀態見狀態總覽