Resolved -
On Monday February 12th, 2024, 03:00 UTC we deployed a code change to a component of Copilot. At 06:00 UTC we observed an increase in timeouts for code completions impacting 55% of Copilot users at peak across Asia and Europe.
At 12:00 UTC we restarted the nodes, and response durations returned to normal operation until 13:00 UTC when response durations degraded again. At 16:15 UTC we made a configuration change to send traffic to regions that were not exhibiting the errors, which resulted in code completions working fully although completing at a higher latency than normal for some users. At 18:00 UTC we reverted the deploy and response durations returned to normal.
We have added better monitoring to components that failed to decrease resolution times to incidents like this in the future.
Feb 12, 18:14 UTC
Update -
Code completion response times have returned to normal.
Feb 12, 18:13 UTC
Update -
We’re still continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Feb 12, 17:30 UTC
Update -
We’re continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Feb 12, 17:01 UTC
Update -
We're continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Feb 12, 16:26 UTC
Update -
Code completions are now working for the impacted users, but completing more slowly than expected. Investigation continues to completely mitigate the issue and restore Copilot code completion functionality to normal.
Feb 12, 15:51 UTC
Update -
Following mitigation steps taken, we have reduced the impact to a more narrow subset of users. Investigation continues to completely mitigate the issue and restore Copilot code completion functionality.
Feb 12, 15:16 UTC
Update -
We are continuing to investigate the issues with Copilot code completion currently impacting some users in Europe. We will provide further details as we have them.
Feb 12, 14:36 UTC
Update -
We have confirmed that this is a reoccurrence of the earlier issue. Impact is currently limited to some European users. The team is working through alternative mitigation strategies to resolve the issue and return normal service.
Feb 12, 14:04 UTC
Update -
We are investigating reports that the earlier problem with Copilot code completions is reoccurring.
Feb 12, 13:28 UTC
Investigating -
We are investigating reports of degraded performance for Copilot
Feb 12, 13:28 UTC
