2.3 KiB
Increased errors for ChatGPT
- 期号: SRE Weekly Issue #462（2025-02-02）
- 作者: OpenAI
- 链接: https://status.openai.com/incidents/k934n9lt39h8
简介
OpenAI posted this mini report on their outage on January 30.
正文
Resolved·Full outage
Affected components
Updates
Write-up published
Resolved
Impact
Between 3:30am PT and 7:08am PT chatgpt.com and a small number of APIs experienced elevated error rates.
Root Cause & Remediation
The root cause of the initial issue was a failure with Cosmos DB in ChatGPT. Our service provider recovered the affected DBs at 4:25am PT.
The second issue was caused by web service pods crash-looping due to their health checks failing, marking pods as unhealthy and subsequently not having enough resources available to service all the requests to the web layer. At this time we believed this was due to users retrying requests. These issues recovered after we implemented a fix to the web server’s health checking logic.
Prevention
- We will make a series of changes to the web systems to more gracefully degrade when they fail including concurrency control and better load shedding.
- A larger investment in making our DB layer more resilient to service provider outages is already underway.
We know that extended outages can impact your products, businesses, and lives. We will prioritize the preventative measures outlined above to continue improving our reliability.
Thu, Jan 30, 2025, 03:57 PM
This issue has now been resolved. Between 4:23am and 7:10am PST, customers experienced elevated errors on ChatGPT
Thu, Jan 23, 2025, 04:05 PM(6 days earlier)
Monitoring
We are continuing to monitor for any further issues.
Thu, Jan 23, 2025, 03:11 PM(54 minutes earlier)
A fix has been implemented and we are monitoring the results.
Thu, Jan 23, 2025, 03:09 PM
Identified
We are continuing to work on a fix for this issue.
Thu, Jan 23, 2025, 02:34 PM(34 minutes earlier)
We have identified the root cause of this issue, and are currently working to implement a fix.
Thu, Jan 23, 2025, 01:43 PM(51 minutes earlier)
Investigating
We are currently experiencing elevated error rates for ChatGPT. We are currently investigating.
Thu, Jan 23, 2025, 01:12 PM(30 minutes earlier)