Resolved -
On July 30th, 2024, between 13:25 UTC and 18:15 UTC, customers using Larger Hosted Runners may have experienced extended queue times for jobs that depended on a Runner with VNet Injection enabled in a virtual network within the East US 2 region. Runners without VNet Injection or those with VNet Injection in other regions were not affected. The issue was caused due to an outage in a third party provider blocking a large percentage of VM allocations in the East US 2 region. Once the underlying issue with the third party provider was resolved, job queue times went back to normal. We are exploring the addition of support for customers to define VNet Injection Runners with VNets across multiple regions to minimize the impact of outages in a single region.
Jul 30, 22:10 UTC
Update -
The mitigation for larger hosted runners has continued to be stable and all job delays are less than 5 minutes. We will be resolving this incident.
Jul 30, 22:09 UTC
Update -
We are continuing to hold this incident open while the team ensures that mitigation put in place is stable.
Jul 30, 21:44 UTC
Update -
Larger hosted runners job starts are stable and starting within expected timeframes. We are monitoring job start times in preparation to resolve this incident. No enqueued larger hosted runner jobs were dropped during this incident.
Jul 30, 21:00 UTC
Update -
Over the past 30 minutes, all larger hosted runner jobs have started in less than 5 minutes. We are continuing to investigate delays in larger hosted runner job starts
Jul 30, 20:17 UTC
Update -
We are still investigating delays in customer’s larger hosted runner job starts. Nearly all jobs are starting under 5 minutes. Only 1 customer larger hosted runner job was delayed by more than 5 minutes in the past 30 minutes.
Jul 30, 19:40 UTC
Update -
We are seeing improvements to the job start times for larger hosted runners for customers. In the last 30 minutes no customer jobs are delayed more than 5 minutes. We will continue monitoring for full recovery.
Jul 30, 19:04 UTC
Update -
We are seeing run delays for larger hosted runners for a limited number of customers. We are deploying mitigations to address these delays.
Jul 30, 18:23 UTC
Investigating -
We are currently investigating this issue.
Jul 30, 18:19 UTC