39 lines
3.6 KiB
Markdown
39 lines
3.6 KiB
Markdown
# Why did AWS Support fail with US-EAST-1 again?
|
||
|
||
- **期号**: SRE Weekly Issue #377（2023-06-18）
|
||
- **作者**: The Stack
|
||
- **链接**: https://www.thestack.technology/us-east-1-aws-support-aws-outage/
|
||
|
||
## 简介
|
||
|
||
AWS had a major Lambda outage in us-east-1, and it took out many customer systems and quite a few other AWS systems, including their support portal.
|
||
|
||
## 正文
|
||
|
||
Amazon has blamed a “subsystem responsible for capacity management for AWS Lambda” for an AWS outage in its US-EAST-1 region that took down over 100 services for approximately four hours late on June 13.
|
||
|
||
Despite recent architectural changes to improve AWS Support resilience AWS said that “customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support” during the incident.
|
||
|
||
The AWS outage took place from 11:49am PDT to 15:37pm PDT.
|
||
|
||
Given its scale, few companies have the ability to affect as many customers in one fell swoop as AWS does; the incident affected FIFA, Fox News, and the McDonalds app amongst many hundreds of others.
|
||
|
||
Some 104 AWS services including AWS Account Management, CloudWatch, Glue, Fargate, Secrets Manager and more saw “increased error rates and latencies” and AWS [said](https://health.aws.amazon.com/health/status?ref=thestack.technology) that “customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS…”
|
||
|
||
The root cause of the issue was rapidly resolved and no cloud hyperscaler (or indeed on-premises data centre) can avoid sporadic issues. The impact on AWS Support that may leave the hyperscaler with some questions to answer, however, after promised changes to AWS Support infrastructure.
|
||
|
||
In late 2021 AWS [promised to build](https://aws.amazon.com/message/12721?ref=thestack.technology) a “new support system architecture that actively runs across multiple AWS regions” amid criticism of US-EAST-1 (AWS’s most fragile region) being a single point of failure for AWS Support when severe [outages like one in December 2021](https://www.thestack.technology/aws-down-december-2021/) happen.
|
||
|
||
This issue was flagged in Gartner’s 2022 Magic Quadrant for Cloud Infrastructure and Platform Services, [which](https://www.gartner.com/doc/reprints?id=1-2BFV0YLG&ct=221019&st=sb&ref=thestack.technology) pointed to AWS’s “regional dependencies and communication” as cause for some concern.
|
||
|
||
### Join peers following [The Stack on LinkedIn](https://www.thestack.technology/linkedin.com/company/stackpublishing)
|
||
|
||
|
||
As Gartner’s analysts put it in October: **“**AWS’s operational incident of 7 December 2021 revealed some multiregion dependencies on the internal AWS network, which is hosted in US-EAST-1. Because US-EAST-1 also hosts support ticketing for North America, AWS customers also had difficulty communicating with technical support during the incident…
|
||
|
||
Yet in a [note](https://aws.amazon.com/about-aws/whats-new/2022/08/aws-support-launches-aws-support-center-console-domain/?ref=thestack.technology) published on August 1, 2022 AWS said it had launched a new “AWS Support Center console URL… [that] ensures you can always contact AWS Support via the AWS Support Center Console… built using the latest architecture standards for high availability and region redundancy.”
|
||
|
||
Why this rebuild did not resolve the [perennial issue of US-EAST-1 outages taking out AWS Support with them](https://www.thestack.technology/aws-support-architecture-service-health-dashboard-criticism-outages/) remains an open question.
|
||
|
||
[*The Stack*](https://thestack.technology/?ref=thestack.technology) has put it to AWS and will update this if we get an answer.
|