Admin API and Customer Runtimes Unavailable

Resolved
Resolved
After 11 hours and 40 minutes

No further issues have been observed within the region. Thank you for your patience as we worked through the various issues presented during this incident. If you continue to encounter issues, please reach out to Kaleido support for further assistance.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 8 hours and 8 minutes

We have identified and resolved the cross-cloud connectivity issues within the region. We are continuing to monitor the situation to ensure that no other issues arise.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 7 hours and 16 minutes

Availability and connectivity within the region have been restored. We are continuing to investigate issues related to cross-cloud connectivity isolated to the region.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 5 hours and 59 minutes

We have completed the secondary steps associated with the restoration of access to existing runtimes and to the creation of new runtimes. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
Recovering
After 5 hours and 23 minutes

We have completed the initial deployment of the fix and have begun to observe recovery within the existing runtimes. Follow-on actions are still being taken to complete the recovery process and we are continuing to monitor the situation. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 4 hours and 38 minutes

We have begun the deployment of the proposed fix. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 53 minutes

We are continuing to test potential resolutions for restoring access to existing runtimes as well as restoring the ability to create new runtimes. We expect our testing to conclude within the next 30 minutes and to being deployment of the proposed fix. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 16 minutes

We are continuing to test potential resolutions for restoring access to existing runtimes as well as restoring the ability to create new runtimes. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 2 hours and 36 minutes

We have completed testing for a resolution to the Admin API and have begun its deployment into production. We have observed marked improvements in this aspect, but acknowledge that this portion of the fix does not resolve the inability to access existing runtimes or to deploy new runtimes.

We are continuing to test potential resolutions for restoring access to existing runtimes as well as restoring the ability to create new runtimes.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 2 hours and 12 minutes

We are continuing to test potential resolutions within our development environments. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 40 minutes

We are continuing to test potential resolutions within our development environments. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
Identified
After 1 hour and 7 minutes

We have identified the issue resulting in timeouts when logging into the Kaleido console and are currently testing potential resolutions within our development environments. We will provide another update in approximately 30 minutes or when more details become available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 33 minutes

No additional details are available at this time. We are continuing to investigate and will provide another update in approximately 30 minutes or when more details become available.

Avatar for
Investigating

We are currently investigating an issue affecting customer runtime and admin API availability in Azure West US 2 region.

Began at:

Affected components
  • Azure US West