6.3 KiB
6.3 KiB
Google Incident report — Meet
- 期号: SRE Weekly Issue #295（2021-11-07）
- 作者: —
- 链接: https://www.google.com/appsstatus/dashboard/incidents/k71P8nHp32hgcMSsC3mR
简介
_Here’s Google’s follow-up report for their October 25-26 Meet outage.
正文
Google Workspace Status Dashboard
contact Support. Learn more about what's posted on the dashboard in
this FAQ. For additional information on these services, please visit
https://workspace.google.com/. For incidents related to Google Analytics, visit the
Incident affecting Google Meet
Incident began at 2021-10-25 11:00 and ended at 2021-10-26 17:00 (times are in Coordinated Universal Time (UTC)).
| Date | Time | Description | |
|---|---|---|---|
| Nov 5, 2021 | 2:27 AM UTC | INCIDENT REPORT Summary The Google Meet Livestream feature experienced disruptions that caused intermittent degraded quality of experience for a small subset of viewers, starting 25 October 2021 0400 PT and ending 26 October 2021 1000 PT. Quality was degraded for a total duration of 4 hours (3 hours on 25 October and 1 hour on 26 October). During this time, no more than 15% of livestream viewers experienced higher rebuffer rates and latency in livestream video playback. We sincerely apologize for the disruption that may have affected your business-critical events. We have identified the cause of the issue and have taken steps to improve our service. Root Cause Our investigation into the incident revealed that the backend message delivery mechanism for Google Meet livestreams that assists with processing experienced an unexpected surge in workload that hit the configuration threshold. As a result, up to 15% of livestream viewers noticed higher rebuffer rates and degraded Meet livestreams. Remediation and Prevention The issue was diagnosed by monitoring some affected viewers that reported degraded performance. Google engineers mitigated the impact by reconfiguring the resources required for processing the affected delivery of data and system messages for higher quality of livestream playbacks. Improvements identified: Increased resource allocation for the backend message delivery system in the short term, and automatically detect message delivery overload in the long term. Enhancements to monitoring systems to capture real time data on quality of livestreams for all volumes of traffic. Alert logic updates to capture spikes in rebuffering rate proactively to help mitigate before any customer impact. Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. Detailed Description of Impact Starting 25 October 2021 0400 PT and ending 26 October 2021 1000 PT, 15% of Google Meet livestream viewers experienced greater than 5% rebuffer rates for a total duration of 4 hours. Viewers with ongoing livestream events were experiencing rebuffering issues. The affected livestreams experienced greater than 5% rebuffering of total viewing time. | |
| Oct 28, 2021 | 11:30 AM UTC | The problem with Google Meet has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. Preliminary Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support . (All Times US/Pacific) Incident Start: 26 October 2021 11:35 Incident End: 27 October 2021 18:00 Duration: 1 day, 6 hours, 25 minutes Affected Services and Features: Google Meet - Livestreams Regions/Zones: Global Description: Google Meet experienced issues for a subset of customers publishing live streams for 1 day, 6 hours, and 25 minutes. From preliminary analysis, the root cause was a configuration change in how livestream media is published to viewers. We will publish a full Incident Report incident once we have completed our internal investigation. Customer Impact: For affected customers' livestreams, a subset of viewers experienced rebuffering or intermittent frame/audio loss. | |
| Oct 28, 2021 | 9:25 AM UTC | Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. | |
| Oct 28, 2021 | 6:39 AM UTC | Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. | |
| Oct 28, 2021 | 3:17 AM UTC | Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. | |
| Oct 28, 2021 | 12:56 AM UTC | Our team is continuing to investigate this issue. We will provide an update by Oct 28, 2021, 3:00 AM UTC with more information about this problem. Thank you for your patience. | |
| Oct 27, 2021 | 10:04 PM UTC | Our team is continuing to investigate this issue. We will provide an update by Oct 28, 2021, 1:00 AM UTC with more information about this problem. Thank you for your patience. | |
| Oct 27, 2021 | 7:04 PM UTC | We are continuing to investigate this issue. We will provide an update by Oct 27, 2021, 10:00 PM UTC detailing when we expect to resolve the problem. | |
| Oct 27, 2021 | 6:05 PM UTC | We're aware of a problem in which Google Meet users may experience latency and/or buffering while viewing livestream events. We will provide an update by Oct 27, 2021, 7:00 PM UTC. | |
| Oct 27, 2021 | 4:58 PM UTC | Google Meet users may experience latency and/or buffering while viewing livestream events. |
- Times are listed in Coordinated Universal Time (UTC)