GitHub header
All Systems Operational
Git Operations ? Operational
API Requests ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Mar 5, 2024

No incidents reported today.

Mar 4, 2024

No incidents reported.

Mar 3, 2024

No incidents reported.

Mar 2, 2024

No incidents reported.

Mar 1, 2024
Incident with API Requests, Copilot, Git Operations, Actions and Pages
Resolved - This incident has been resolved.
Mar 1, 17:42 UTC
Update - Git Operations is operating normally.
Mar 1, 17:42 UTC
Update - Actions and Pages are operating normally.
Mar 1, 17:41 UTC
Update - Copilot is operating normally.
Mar 1, 17:36 UTC
Update - Pages is experiencing degraded performance. We are continuing to investigate.
Mar 1, 17:34 UTC
Update - One of our clusters is experiencing problems, and we are working on restoring the cluster at this time.
Mar 1, 17:34 UTC
Investigating - We are investigating reports of degraded performance for API Requests, Copilot, Git Operations and Actions
Mar 1, 17:30 UTC
Incident with Pull Requests, Actions and Issues
Resolved - This incident has been resolved.
Mar 1, 16:12 UTC
Update - Issues, Pull Requests and Actions are operating normally.
Mar 1, 16:12 UTC
Update - We're seeing our background job queue sizes trend down, and expect full recovery in the next 15 minutes.
Mar 1, 15:48 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.
Mar 1, 15:39 UTC
Update - We're continuing to investigate issues with background jobs that have impacted Actions and Pull Requests. We have a mitigation in place and are monitoring for recovery.
Mar 1, 15:27 UTC
Update - We're investigating issues with background jobs that are causing sporadic delays in pull request synchronization and reduced Actions throughput.
Mar 1, 14:51 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests and Actions
Mar 1, 14:39 UTC
Feb 29, 2024
Incident with Issues, Webhooks and Actions
Resolved - On February 29, 2024, between 9:32 and 11:54 UTC, queuing in our background job service caused processing delays to Webhooks, Actions, and Issues. Nearly 95% of delays occurred between 11:05 and 11:27 UTC, with 5% during the remainder of the incident. During this incident, the following customer impacts occurred: 50% of webhooks experienced delays of up to 5m, 1% of webhooks experienced delays of 17m at peak; Actions: on average, 7% of customers experienced delays, with a peak of 44%; and many Issues saw a delay in appearing in searches. At 9:32 UTC our automated failover successfully routed traffic to a secondary cluster. But an improper restoration to primary at 10:32 UTC caused a significant increase in queued jobs until 11:21 UTC, when a correction was made and healthy services began burning down the backlog until full resolution.

We have made improvements to the automation and reliability of our fallback process to prevent recurrence. We also have larger work already in progress to improve the overall reliability of our job processing platform.
Feb 29, 12:27 UTC
Update - We're seeing recovery and are going to take time to verify that all systems are back in a working state.
Feb 29, 12:21 UTC
Update - Issues is operating normally.
Feb 29, 12:19 UTC
Update - Webhooks is operating normally.
Feb 29, 12:18 UTC
Update - We're continuing to investigate delayed background jobs. We've seen partial recovery for Issues, and there is ongoing impact to actions, notifications and webhooks.
Feb 29, 11:05 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Feb 29, 10:58 UTC
Update - We're seeing issues related to background jobs, which are causing delays for webhook delivery and search indexing, and other updates.
Feb 29, 10:36 UTC
Investigating - We are investigating reports of degraded performance for Issues and Webhooks
Feb 29, 10:33 UTC
Feb 28, 2024

No incidents reported.

Feb 27, 2024

No incidents reported.

Feb 26, 2024
We are investigating reports of degraded performance.
Resolved - On Monday, February 26th, from 20:45 UTC to 21:39 UTC, GitHub Packages reported an outage indicating a degradation in GitHub Container Registry and NPM package upload functionality. Upon investigation, we found a misconfigured observability metric which inadvertently pulled in data from a newly provisioned test environment. All failures being reported were traced back to this test environment. We confirmed that there was no real customer impact to GitHub Packages during this incident. We have since reconfigured our observability metrics to accurately report based on environment.
Feb 26, 21:40 UTC
Update - We are seeing some recovery in NPM and GitHub Container Registry functionality, but are maintaining red status until we are certain issues won’t recur.
Feb 26, 21:20 UTC
Update - NPM and GitHub Container Registry services are degraded, particularly the upload functionality. Investigation is underway.
Feb 26, 21:03 UTC
Investigating - We are currently investigating this issue.
Feb 26, 21:01 UTC
Incident with Webhooks, Actions, Pull Requests and Issues
Resolved - On February 26, 2024, between 18:34 UTC and 19:37 UTC our background job service was degraded and caused job start delays up to 15 minutes. Users experienced delays in Webhooks, Actions, and some UI updates (e.g. a delay in UI updates on pull requests). This was due to capacity problems with our job queueing service, and a failure of our automated failover system.

We mitigated the incident by manually failing over to our secondary cluster. No data was lost - recovery began at 18:55 UTC, when the backlog of enqueued jobs began to process.

We are actively working to repair our failover automation and expand the capacity of our background job queuing service to prevent issues like this in the future.
Feb 26, 19:37 UTC
Update - Actions and Pull Requests are operating normally.
Feb 26, 19:37 UTC
Update - Webhooks and Issues are operating normally.
Feb 26, 19:37 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.
Feb 26, 19:05 UTC
Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Feb 26, 18:57 UTC
Update - We have deployed a fix for issues affecting Webhooks, Actions, and some other services. We are beginning to see recovery and will continue to monitor and fix as needed.
Feb 26, 18:55 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Feb 26, 18:55 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Feb 26, 18:48 UTC
Investigating - We are investigating reports of degraded performance for Webhooks
Feb 26, 18:47 UTC
Feb 25, 2024

No incidents reported.

Feb 24, 2024

No incidents reported.

Feb 23, 2024

No incidents reported.

Feb 22, 2024

No incidents reported.

Feb 21, 2024
Incident with Actions
Resolved - On Wednesday February 21, 2024, 17:07 UTC, we deployed a configuration change to one of our services inside of Actions. At 17:14 UTC we noticed an increase in exceptions that impacted approximately 85% of runs at that time.

At 17:18 UTC, we reverted the deployment and our service immediately recovered. During this timeframe, customers may have noticed their workflows failed to trigger or workflows were queued but did not progress.

To prevent this issue in the future we are improving our deployment observability tooling to detect errors earlier in the deployment pipeline.
Feb 21, 17:30 UTC
Investigating - We are investigating reports of degraded performance for Actions
Feb 21, 17:20 UTC
Feb 20, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage