Resolved -
On 2023-09-19 at 20:36 UTC, while migrating the primary datastore for Projects, we caused an outage that led to Projects data becoming unavailable for approximately four hours.
While working to restore Projects data, we also experienced a data replication interruption which affected Git Operations, APIs, and Issues. We first resolved the data replication issue, which returned Git Operations, APIs, and Issues to normal operation. We then restored Projects data to its pre-migration state, and re-inserted data that was added during the period of partial availability. The incident was resolved on 2023-09-20 at 00:06 UTC.
As a result of this incident, we have improved validation of data migrations in test and during rollout. We have also identified improvements to reduce both the time to restore and to fix the replication issue.
Sep 20, 04:28 UTC
Update -
Issues is operating normally.
Sep 20, 04:02 UTC
Update -
We experienced an incident causing missing Project item data. We have restored all data from before the incident, up to 19 Sep 20:36 UTC. Project data modified between 20:36 UTC and 20 Sep 00:06 UTC was partially restored.
We will publish a root cause analysis for this issue by Sep 29.
Sep 20, 04:01 UTC
Update -
Projects data has been restored to the state prior to the incident (19 Sep 20:57 UTC). We are continuing to restore the last of Project metadata updated since 20:57 UTC.
Sep 20, 03:17 UTC
Update -
Projects data has been restored to the state prior to the incident (19 Sep 20:57 UTC). We are continuing to restore the last of Project metadata updated since 20:57 UTC.
Sep 20, 02:35 UTC
Update -
While rolling out an update to Projects, Project data was affected and needed to be restored. While restoring data, we experienced a replication interruption which affected Git Operations, Issues and APIs. We restarted replication, and restored Projects data to its state prior to the rollout, and continue working to restore the last of Project metadata updated since 19 Sep 20:57 UTC.
Sep 20, 01:53 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Sep 20, 00:44 UTC
Update -
We've restored access to a majority of Project data. A small number of customers are still affected, and we continue to work on full restoration.
Sep 20, 00:20 UTC
Update -
We continue restoring access to Project data. We do not have an estimated time to full restoration at this time, and are working to provide that.
Sep 20, 00:04 UTC
Update -
We continue working on restoring access to Projects data, along with an estimated time to full restoration.
Sep 19, 23:33 UTC
Update -
API Requests is operating normally.
Sep 19, 23:00 UTC
Update -
Git Operations is operating normally.
Sep 19, 22:59 UTC
Update -
Git operations, API requests, and Issues are recovering after restoring data replication, and we continue working on restoring access to Projects data.
Sep 19, 22:55 UTC
Update -
Git Operations is experiencing degraded performance. We are continuing to investigate.
Sep 19, 22:23 UTC
Update -
* We are actively working on restoring access to Projects data.
* Issues and APIs are experiencing degraded performance. We are working on mitigating the issue.
Sep 19, 22:21 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Sep 19, 21:48 UTC
Update -
We have identified the issue and are actively working to restore missing project items
Sep 19, 21:29 UTC
Update -
Issues is experiencing degraded availability. We are continuing to investigate.
Sep 19, 21:10 UTC
Update -
We are currently investigating an issue with project items not loading
Sep 19, 21:01 UTC
Investigating -
We are currently investigating this issue.
Sep 19, 20:57 UTC
