4.9 KiB
Supermarket Berkshelf Incident Post Mortem
- 期号: SRE Weekly Issue #25（2016-05-29）
- 作者: —
- 链接: https://www.chef.io/blog/2016/05/26/supermarket-berkshelf-incident-post-mortem/
简介
This blows my mind. Chef held a live, public retrospective meeting for a recent production incident. I love this idea and I can only hope that more companies follow suit. The transparency is great, but more than that is their sharing of their retrospective process itself. They have a well-defined format for retrospectives including a statement of blamelessness at the beginning. Kudos to Chef for this, and thanks to Nell Shamrell-Harrington for posting the link on Hangops.
The actual incident was fairly interesting too. The crux of it comes down to this quote that we’ve probably all uttered ourselves at one point or another:
The further distant staging is from production, the more likely we are to introduce a bug.
正文
We at Chef believe it is important to conduct public post mortems whenever possible. We recently conducted one around a Supermarket/Berkshelf incident that occurred on May 16, 2016. I was the incident commander for this incident and would like to share both the video and write up.
Video Recording
Write Up
Description
On May 16 we experienced a brief SSL issue between Supermarket and Berkshelf.
Timeline
This incident began at 21:56UTC on Monday, May 16, 2016. It was resolved at 22:49UTC that same day.
**Time to detect**: 13 minutes 21:56UTC - 22:09UTC on Monday, May 16, 2016
**Time to resolve**: 44 minutes 21:56UTC - 22:36UTC on Monday, May 16, 2016
All times UTC
21:56 - Nell Shamrell-Harrington upgraded 2 of the 4 Supermarket Prod nodes from Supermarket 2.5.2 to Supermarket 2.6.0. She also upgraded the cookbook versions of oc-omnibus-supermarket and supermarket-omnibus-cookbook
22:09 - Nell Shamrell-Harrington ran berks install to pull cookbooks from the public Supermarket and received this error:
OpenSSL::SSL::SSLError: hostname "community-files.opscode.com.s3.amazonaws.com" does not match the server certificate
She asked in the internal Chef Slack if someone else would run berks install to confirm what she was seeing
22:24 - Lamont Grandquist confirmed that he was seeing the same error in Travis builds
22:32 - Nell Shamrell-Harrington declared an incident
22:36 - Nell Shamrell-Harrington moved the two upgraded Supermarket prod nodes out of the Supermarket prod ELB and confirmed that she no longer saw the error when running berks install
22:38 - SaintAardvark in the #chef IRC channel reported SSL issues with running Berks install, Noah Katrowitz mentioned that kisoku (#chef IRC handle) was reporting the same thing
22:39 - Noah Kantrowitz DM'd Nell Shamrell-Harrington to let her know that users in the Chef IRC channel were reporting issues with berks and Supermarket
22:43 - Lamont Grandquist reported that Travis runs were working again
22:46 - Nell Shamrell-Harrington entered #chef IRC
22:47 - kisoku reported that his CI jobs were working again in #chef IRC
22:50 - SaintAardvark reported that his Jenkins jobs were working again in #chef IRC
22:49 - Nell Shamrell-Harrington declared the incident closed
Contributing Factor(s)
The 2.6.0 release of Supermarket included a commit which changed the AWS S3 urls used to access cookbook artifacts in S3 storage. Prior to this change, Supermarket (through the Paperclip plug in) used a hosted-style S3 url. The one for public Supermarket looked like this: https://s3.amazonaws.com/community-files.opscode.com/
The problem was this URL style only worked if an S3 bucket was in N. Virginia. To fix this, we changed our config to use a path-style url like this: https://community-files.opscode.com.s3.amazonaws.com
When this change was merged and deployed, this error appeared when someone attempted to do a berks install using public Supermarket as the cookbook source:
OpenSSL::SSL::SSLError: hostname "community-files.opscode.com.s3.amazonaws.com" does not match the server certificate
This was due to there being “.” in the bucket name “community-files.opscode.com.s3.amazonaws.com.” Although the previous S3 url style worked with dots in the bucket name, it did not work for a path-style url
Stabilization Steps
We had fortunately only upgraded 2 of the 4 prod nodes, so we removed the 2 upgraded nodes from the ELB, then downgraded them back to Supermarket 2.5.2
Impact
For approximately 53 minutes, anyone using berks install saw the SSL error.
Corrective Actions
- Make S3 url style configurable in Supermarket
- Make sure staging bucket has similar formatted name to the production bucket
- Ensure that berks install is part of smoke tests in both staging and production
- Add documentation around considerations when naming an S3 bucket
- Investigate adding a monitor that does a simple berks install and executes on a regular basis