与GitHub.com的事件

2026年8月17日，UTC时间13:28至21:15（7小时47分钟），GitHub.com在Issues、Pull Requests、APIs、Actions和Copilot方面经历了错误率升高和延迟增加。高峰期，Web/API错误率约为20%，而归档和原始内容下载的错误率达到了约50%。SAML/OIDC认证、SCIM和Team Sync也受到影响，以及GHEC中依赖托管在GitHub.com上的公共工作流步骤定义且具有数据驻留功能的Actions工作流。随着我们美国中部数据中心的恢复，大多数服务在UTC时间16:36前恢复正常；Actions服务降级持续至约18:03 UTC；Copilot Token Service在21:02完全恢复。<br /><br />部分故障流量从美国中部转移至北弗吉尼亚，并在那里成功处理，直到美国中部的网络故障被调试并解决。对单个内部端点的延迟响应触发了VS Code中一个潜在的重试缺陷，导致流量放大约10倍，并延迟了Copilot Token Service的恢复。<br /><br />故障的直接原因是由新流量峰值导致的美国中部负载均衡器网络饱和。最初，一个Istio sidecar pod达到其并发限制，并因配置错误的策略（监控主机服务而非sidecar限制）未能正确自动扩展。一个故障级联至更多故障，最终四个HAProxy节点耗尽了其流限制，导致网关认证路径降级，引发广泛的认证延迟和失败。乐观重试逻辑使问题恶化，过载了内部负载均衡器。同时暂停这些节点上的HAProxy立即带来了广泛的恢复。<br /><br />北弗吉尼亚的重试风暴通过以下方式解决：1）通过PR暂时减少网关重试逻辑；2）在负载均衡器上以403阻止入站的Copilot Token Service令牌请求，然后逐步按站点恢复流量，使调用方能够成功。<br /><br />残余的Copilot认证失败持续存在，因为客户端重试行为放大了负载：一次失败的令牌操作可能产生大量额外请求并进入重试循环。Copilot Token Service流量从正常的7-9K RPS增加到70-100K RPS。减少网关认证重试并阻止触发重试的响应稳定了Copilot Token Service并完成了恢复。<br /><br />阻碍恢复的复杂因素包括对codeload端点的大量爬取攻击。<br /><br />为防止再次发生，我们的后续行动包括：<br /><br />- 修正自动扩展策略，以考虑服务网格sidecar的并发性和容量。<br /><br />- 审计受影响服务的Istio请求、并发和扩展限制。<br /><br />- 审查网关和客户端的重试限制及退避行为。<br /><br />- 解决放大Copilot令牌流量的VS Code重试行为。<br /><br />- 改进负载均衡器容量监控和区域故障转移保障措施。

原文On August 17, 2026, from 13:28–21:15 UTC (7h 47m), GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02. <br /><br />Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved. Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service. <br /><br />The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. <br /><br />The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed. <br /><br />Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery. <br /><br />Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints. <br /><br />To prevent recurrence, our follow-up actions include: <br /><br />- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity. <br /><br />- Auditing Istio request, concurrency, and scaling limits across affected services. <br /><br />- Reviewing retry limits and backoff behavior across gateways and clients. <br /><br />- Addressing the VS Code retry behavior that amplified Copilot token traffic. <br /><br />- Improving load-balancer capacity monitoring and regional failover safeguards.

我们正在继续应用缓解措施，以解决某些应用程序中偶发的Copilot身份验证失败问题。预计将在接下来的30分钟内完全恢复。通过GitHub CLI和GitHub App使用Copilot不受影响。

原文We are continuing to apply mitigations to address sporadic Copilot authentication failures in some applications. We expect full recovery within the next 30 minutes. Copilot usage via the GitHub CLI and GitHub App are unaffected.

问题正在正常运行。

原文Issues is operating normally.

我们正在继续调查影响某些应用程序中Copilot认证的偶发性故障。通过GitHub CLI和GitHub App使用Copilot不受影响。

原文We are continuing to investigate sporadic failures affecting Copilot authentication in some applications. Copilot usage via the GitHub CLI and GitHub App are unaffected.

我们正在继续调查偶发的身份验证失败问题。我们已部分禁用身份验证令牌重试机制，并观察到情况有所改善，目前正在监控影响，之后将全面应用此缓解措施。

原文We are continuing to investigate sporadic authentication failures. We have partially disabled authentication token retries and have seen improvement, and we are monitoring impact before fully applying this mitigation.

API Requests 运行正常。

原文API Requests is operating normally.

API Requests 正经历可用性降级。我们正在继续调查。

原文API Requests is experiencing degraded availability. We are continuing to investigate.

影响Git操作的服务降级已得到缓解。我们正在监控以确保稳定性。

原文The degradation affecting Git Operations has been mitigated. We are monitoring to ensure stability.

我们已定位到问题组件并采取了纠正措施，但仍观察到零星认证失败形式的残余影响。我们正在继续应用额外的缓解措施，并调查剩余影响。

原文We identified the problematic component and have taken corrective actions, but we are seeing residual impact in the form of sporadic authentication failures. We are continuing to apply additional mitigations and investigate the remaining impact.

服务目前性能下降。我们正在继续调查。

原文Issues is experiencing degraded performance. We are continuing to investigate.

我们已定位问题组件并采取了纠正措施，但众多服务仍受到残余影响。我们正在继续实施额外的缓解措施，并调查剩余影响。

原文We identified the problematic component and have taken corrective actions, but we are seeing residual impact across numerous services. We are continuing to apply additional mitigations and investigate the remaining impact.

Git Operations 正在经历性能下降。我们正在继续调查。

原文Git Operations is experiencing degraded performance. We are continuing to investigate.

影响API请求、操作、Git操作、问题、页面、拉取请求和Webhooks的性能下降已得到缓解。我们正在监控以确保稳定性。

原文The degradation affecting API Requests, Actions, Git Operations, Issues, Pages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.

我们已定位到问题组件并采取了纠正措施。目前有强烈的恢复迹象，但我们仍在努力完全恢复服务，错误率仍略高于正常水平。随着恢复工作的继续，我们将发布进一步更新。

原文We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues.

我们正在经历约20%的高错误率，影响网页体验和API流量。归档下载和原始仓库内容下载的错误率约为50%。SAML和OIDC认证、SCIM以及Team Sync也受到影响。我们仍在努力确定根本原因，并将随着进一步了解和采取缓解措施继续发布更新。

原文We are experiencing high error rates around 20% for web experiences and api traffic. Archive downloads and raw repository content downloads are experiencing an approximate 50% error rate. SAML and OIDC authentication, SCIM, and Team Sync are also impacted. We are still working to identify the root cause and will continue to post updates as we learn more and perform mitigation.

我们正在经历约20%的高错误率，影响网页体验和API流量。归档下载和原始仓库内容下载的错误率约为50%。SAML和OIDC认证、SCIM以及Team Sync也受到影响。我们目前正在进行缓解措施，并将随着进展发布更新。

原文We are experiencing high error rates around 20% for web experiences and api traffic. Archive downloads and raw repository content downloads are experiencing an approximate 50% error rate. SAML and OIDC authentication, SCIM, and Team Sync are also impacted. We are currently performing mitigations and will post updates as we progress.

Webhooks 目前性能下降。我们正在继续调查。

原文Webhooks is experiencing degraded performance. We are continuing to investigate.

Git Operations 正在经历性能下降。我们正在继续调查。

原文Git Operations is experiencing degraded performance. We are continuing to investigate.

Pages 正在经历性能降级。我们正在继续调查。

原文Pages is experiencing degraded performance. We are continuing to investigate.

API Requests 正经历可用性降级。我们正在继续调查。

原文API Requests is experiencing degraded availability. We are continuing to investigate.

Webhooks 服务目前遇到可用性降级问题。我们正在继续调查中。

原文Webhooks is experiencing degraded availability. We are continuing to investigate.

我们目前正经历约20%的高错误率，影响网页体验和API流量。归档下载和原始仓库内容下载的错误率约为50%。SAML和OIDC认证、SCIM及Team Sync也受到影响。我们正根据目前的调查结果进行缓解措施，并持续监控改善情况。

原文We are experiencing high error rates around 20% for web experiences and api traffic. Archive downloads and raw repository content downloads are experiencing an approximate 50% error rate. SAML and OIDC authentication, SCIM, and Team Sync are also impacted. We are currently performing mitigations based on our investigation thus far and are monitoring for improvement.

Actions 正经历可用性降级。我们正在继续调查。

原文Actions is experiencing degraded availability. We are continuing to investigate.

Pull Requests 正经历可用性降级。我们正在继续调查。

原文Pull Requests is experiencing degraded availability. We are continuing to investigate.

服务正在经历可用性降级。我们正在继续调查。

原文Issues is experiencing degraded availability. We are continuing to investigate.

Pull Requests 正经历可用性降级。我们正在继续调查。

原文Pull Requests is experiencing degraded availability. We are continuing to investigate.

Copilot 正在经历可用性降级。我们正在继续调查。

原文Copilot is experiencing degraded availability. We are continuing to investigate.

我们正在经历约20%的高错误率，影响网页体验和API流量。归档下载和原始仓库内容下载的错误率约为50%。SAML和OIDC认证、SCIM以及Team Sync也受到影响。调查正在进行中，我们将继续在发现更多信息时提供更新。

原文We are experiencing high error rates around 20% for web experiences and api traffic. Archive downloads and raw repository content downloads are experiencing an approximate 50% error rate. SAML and OIDC authentication, SCIM, and Team Sync are also impacted. Investigations are on-going and we will continue to provide updates as we discover more information.

我们正在经历约20%的高错误率，影响网页体验和API流量。归档下载和原始仓库内容下载的错误率约为50%。目前正在持续调查根本原因，调查过程中我们将继续提供更新。

原文We are experiencing high error rates around 20% for web experiences and api traffic. Archive downloads and raw repository content downloads are experiencing an approximate 50% error rate. Investigations are on-going into the root cause, and updates will continue to be provided as we investigate.

Pull Requests 正经历性能下降。我们正在继续调查。

原文Pull Requests is experiencing degraded performance. We are continuing to investigate.

服务目前性能下降。我们正在继续调查。

原文Issues is experiencing degraded performance. We are continuing to investigate.

我们观察到包括Pull Requests、Issues等多项体验的错误率约为20%。目前正在调查中，如有更新我们将及时发布。

原文We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available

Webhooks 目前性能下降。我们正在继续调查。

原文Webhooks is experiencing degraded performance. We are continuing to investigate.

Actions 正在经历性能降级。我们正在继续调查。

原文Actions is experiencing degraded performance. We are continuing to investigate.

API Requests 正经历性能降级。我们正在继续调查。

原文API Requests is experiencing degraded performance. We are continuing to investigate.

我们正在调查部分GitHub服务性能受影响的相关报告。

原文We are investigating reports of impacted performance for some GitHub services.