多个服务发生故障

自 9 月 23 日 07:57 UTC 起，GitHub 在多个应用页面上出现了 500 和 404 响应增多的情况。这导致安装 GitHub Apps、创建组织以及进行某些组织成员变更时失败。客户还遇到了标签更新延迟以及 Projects 中搜索结果陈旧的问题。<br /><br />此次基础设施故障仅限于我们的 Azure 美国中部区域。API 错误已于 9 月 23 日 10:58 UTC 得到缓解。Projects 的处理在积压任务被逐步清空的同时继续恢复，完整服务已于 9 月 24 日 04:55 UTC 恢复。<br /><br />该事件是由一次对主数据库的计划内维护操作失败引起的。自动恢复启动了紧急数据库故障转移，此后受影响区域中的多个副本无法正确恢复复制。这减少了可用的数据库容量，并导致了 API 错误和下游 Projects 处理延迟。<br /><br />我们已经缓解了直接的故障模式。我们还在改进维护安全检查、数据库故障转移处理、故障转移后的副本验证以及下游处理韧性，以降低类似事件发生的可能性和影响。

原文Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects. <br /><br />The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects' processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24. <br /><br />The incident was caused by a failed planned maintenance operation on a primary database. Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly. This reduced available database capacity and caused the API errors and downstream Projects processing delays. <br /><br />We have mitigated the immediate failure mode. We are also improving maintenance safety checks, database failover handling, post-failover replica validation, and downstream processing resilience to reduce the likelihood and impact of similar incidents.

故障已缓解。我们正在监控以确保稳定性。

原文The degradation has been mitigated. We are monitoring to ensure stability.

我们正在继续处理Projects中积压的问题标签更新。标签更改可能仍会延迟。所有其他服务均正常运行。

原文We are continuing to process the backlog of issue label updates for Projects. Label changes may still be delayed. All other services are operating normally.

我们正在继续处理Projects中问题标签更新的积压工作。用户可能仍会遇到标签更改在Projects中反映之前的延迟。所有其他服务均正常运行。

原文We are continuing to process the backlog of issue label updates for Projects. Users may still see delays before label changes are reflected in Projects. All other services are operating normally.

我们已部署了一项变更，旨在加速处理Projects中积压的问题标签更新。目前仍有大量积压，我们正持续努力处理。所有其他服务均正常运行。我们将在接下来的一小时内提供进一步更新。

原文We've deployed a change intended to accelerate processing of the backlog of issue label updates in Projects. A sizable backlog still remains and we continue working through it. All other services are operating normally. We will provide another update within the next hour.

问题标签的更新在 Projects 中可能会延迟显示。我们正在持续部署一项变更，以加快处理积压的标签更新。所有其他服务均可用。我们将在接下来的一小时内提供另一条更新。

原文Updates to issue labels may be delayed in being reflected in Projects. We are continuing to deploy a change that will accelerate processing of the backlog of label updates. All other services are available. We will provide another update within the next hour.

我们正准备部署一项变更以减轻影响。

原文We are preparing to deploy a change that will mitigate the impact.

继续调查问题标签在项目中准确反映时可能出现的延迟。我们正在研究替代方案，以处理积压的标签更新。

原文Continuing to investigate the lag that may be experienced in issue labels being accurately reflected in Projects. We are working on alternate solutions to process the backlog of label updates.

我们将在大约一小时后发布另一条更新，分享我们的进展。

原文We will post another update in approximately one hour to share our progress.

问题标签的更新在 Projects 中反映可能会有大约 10 分钟的延迟。我们已增加了一些容量以更快地处理积压，但完成全部积压消息的处理可能还需要几个小时。所有其他服务均可用。

原文Updates to issue labels may be delayed in being reflected in Projects by about ~10 minutes. We have added some capacity to work through the backlog more quickly, but it'll likely be a few hours to complete processing the full backlog of messages. All other services are available.

用户可能会遇到项目搜索结果陈旧的问题。我们正在努力提升索引速度。其他所有服务均可用。

原文Users may experience stale Project search results. We are working to increase indexing speed. All other services are available.

我们看到 Projects 正在恢复。在索引追赶期间，用户可能会遇到 Projects 搜索结果陈旧的情况。

原文We are seeing recovery for Projects. Users may experience stale search results for Projects while indexing catches up.

影响API请求的服务降级问题已得到缓解。我们正在持续监控以确保稳定性。

原文The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.

数据库副本已恢复。组织创建和GitHub API不再降级。

原文Database replicas have been restored. Org creation and the GitHub API are no longer degraded.

数据库副本已分离。我们正在努力恢复数据库副本。用户可能会遇到创建组织之外的问题，以及GitHub API和Projects体验下降的情况。

原文Database replicas have detached. We're working to restore the database replicas. Users may experience issues beyond creating organizations and a degraded experience with the GitHub API and Projects.

我们正在调查有关API请求性能下降的报告。

原文We are investigating reports of degraded performance for API Requests