83 lines
5.8 KiB
Markdown
83 lines
5.8 KiB
Markdown
# Christmas Come Early: An AWS EBS Performance Regression Update
|
||
|
||
- **期号**: SRE Weekly Issue #402（2023-12-10）
|
||
- **作者**: Dustin Brown — dolthub
|
||
- **链接**: https://www.dolthub.com/blog/2023-12-08-christmas-come-early-ebs-performance-regression-update/
|
||
|
||
## 简介
|
||
|
||
More on the weird EBS performance regression I linked to last week. Still no full explanation of what changed, but at least they have a solution (gp3 volumes).
|
||
|
||
## 正文
|
||
|
||
Two weeks ago we published [A Spooky Performance Regression in AWS EBS Volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes.html) that highlighted a performance regression in writes against AWS EBS volumes we saw while benchmarking [Dolt](https://www.doltdb.com) and [MySQL](https://www.mysql.com).
|
||
|
||
Initially we suspected that a recent self-imposed change to our benchmarking infrastructure resulted in the write performance slow-down, but after numerous tests and sanity checks, we concluded that something outside of our control had changed with the AWS EBS volumes we were using.
|
||
|
||
Well, today we have an update—a Christmas update! (Since Thanksgiving has come and gone).
|
||
|
||
|
||
|
||
|
||
Soon after our previous blog went out, a user reached out on Discord with this comment:
|
||
|
||
|
||
|
||
|
||
For benchmarking Dolt’s performance, we’ve always used a 30GB `gp2` volume. `gp2` volumes are general purpose solid state drives (SSDs), which guarantee a minimum of 100 input/output operations per second (IOPS). For these volume types, the IOPS scales linearly with volume size, so if we increased the size of the volume, we’d also increase the provisioned IOPS.
|
||
|
||
In 2020, AWS released `gp3` volumes, a new SSD type that decouples IOPS from volume size allowing users to provision the required performance without provisioning additional block storage capacity. Contrary to `gp2` volumes, `gp3` provisions a minimum of 3000 IOPS, regardless of volume size. More, of course, can be provisioned, but the volume size does not need to increase in tandem.
|
||
|
||
So, we went ahead and decided to give the `gp3` volumes a shot. The result was the most wonderful, early Christmas gift us Dolt’s could have asked for—our Sysbench write performance regression was gone!
|
||
|
||
|
||
|
||
|
||
On the `gp2` backed benchmarking hosts we saw MySQL perform worse than Dolt on multiple benchmarks (though both servers saw increased write latency, MySQL’s write latency increased significantly more):
|
||
|
||
| write_tests | server_name | server_version | latency_median | server_name | server_version | latency_median | multiplier |
|
||
|---|---|---|---|---|---|---|---|
|
||
| oltp_delete_insert | mysql | 8.0.35 | 7.98 | dolt | c7b85e8c | 6.79 | 0.9 |
|
||
| oltp_insert | mysql | 8.0.35 | 3.75 | dolt | c7b85e8c | 3.36 | 0.9 |
|
||
| oltp_read_write | mysql | 8.0.35 | 8.43 | dolt | c7b85e8c | 14.46 | 2.1 |
|
||
| oltp_update_index | mysql | 8.0.35 | 3.82 | dolt | c7b85e8c | 3.36 | 0.9 |
|
||
| oltp_update_non_index | mysql | 8.0.35 | 3.82 | dolt | c7b85e8c | 3.3 | 0.9 |
|
||
| oltp_write_only | mysql | 8.0.35 | 5.37 | dolt | c7b85e8c | 7.56 | 1.4 |
|
||
| types_delete_insert | mysql | 8.0.35 | 7.7 | dolt | c7b85e8c | 7.3 | 1.0 |
|
||
|
||
Each `multiplier` of `0.9` means that Dolt was faster on `gp2` than MySQL was, but this didn’t match our expectation. At best, we expect Dolt to be equally as fast as MySQL.
|
||
|
||
On `gp3` backed hosts, though, we now see write latency down across both servers to pre-Halloween levels, which is what we expect to see from our benchmarking tests:
|
||
|
||
| write_tests | server_name | server_version | latency_median | server_name | server_version | latency_median | multiplier |
|
||
|---|---|---|---|---|---|---|---|
|
||
| oltp_delete_insert | mysql | 8.0.35 | 5.57 | dolt | 1.29.2 | 5.99 | 1.1 |
|
||
| oltp_insert | mysql | 8.0.35 | 2.81 | dolt | 1.29.2 | 2.91 | 1.0 |
|
||
| oltp_read_write | mysql | 8.0.35 | 7.3 | dolt | 1.29.2 | 14.73 | 2.0 |
|
||
| oltp_update_index | mysql | 8.0.35 | 2.86 | dolt | 1.29.2 | 3.02 | 1.1 |
|
||
| oltp_update_non_index | mysql | 8.0.35 | 2.97 | dolt | 1.29.2 | 3.02 | 1.1 |
|
||
| oltp_write_only | mysql | 8.0.35 | 4.1 | dolt | 1.29.2 | 7.17 | 1.7 |
|
||
| types_delete_insert | mysql | 8.0.35 | 5.47 | dolt | 1.29.2 | 6.55 | 1.2 |
|
||
|
||
As a result, we’ve updated our benchmarking infrastructure to now provision `gp3` EBS volumes for benchmarking Dolt, and we’ll keep an eye out for any other changes to Dolt’s performance, especially if it’s unexpected.
|
||
|
||
# Conclusion[#](https://www.dolthub.com#conclusion)
|
||
|
||
Big shout out to `fernomac` for the early `gp3` Christmas gift.
|
||
|
||
Fixing our benchmarking infrastructure only required a one line change to our Terraform code, so that was nice too.
|
||
|
||
We love having users and blog readers reach out to us on [Discord](https://discord.gg/gqr7K4VNKe) so come by and chat with us about how you’re using Dolt.
|
||
|
||
We still aren’t sure what happened with `gp2` EBS volumes on Halloween, but if you’re experiencing any slowness we’d highly recommend migrating to `gp3` as well.
|
||
|
||
|
||
|
||
|
||
Happy Holidays and don’t forget to check out each of our different product offerings below, to find which ones are right for you:
|
||
|
||
- [Dolt](http://www.doltdb.com) —it’s Git for data.
|
||
- [DoltHub](https://www.dolthub.com) —it’s GitHub for Dolt.
|
||
- [DoltLab](https://www.doltlab.com) —it’s GitLab for data.
|
||
- [Hosted Dolt](https://hosted.doltdb.com/) —it’s RDS for Dolt databases.
|