198 lines
9.1 KiB
Markdown
198 lines
9.1 KiB
Markdown
# fluxninja/aperture: Flow control and reliability management for modern cloud applications
|
||
|
||
- **期号**: SRE Weekly Issue #339（2022-09-18）
|
||
- **作者**: FluxNinja
|
||
- **链接**: https://github.com/fluxninja/aperture
|
||
|
||
## 简介
|
||
|
||
Aperture is a new open source tool helps you prevent cascading failures using load-shedding and rate limiting.
|
||
|
||
BONUS CONTENT: Here‘s their article explaining how it works.
|
||
|
||
## 正文
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
Aperture is a distributed load management platform designed for rate limiting, caching, and prioritizing requests in cloud applications. Built upon a foundation of distributed counters, observability, and a global control plane, it provides a comprehensive suite of load management capabilities. These capabilities enhance the reliability and performance of cloud applications, while also optimizing cost and resource utilization.
|
||
|
||
|
||
Integrating Aperture in your application through SDKs is a simple 3-step process:
|
||
|
||
- **Define labels** : Define labels to identify users, entities, or features
|
||
within your application. For example, you can define labels to identify
|
||
individual users, features, or API endpoints.
|
||
|
||
## Example
|
||
|
||
```
|
||
// Tailor policies to get deeper insights into your workload with labels that
|
||
// capture business context.
|
||
const labels = {
|
||
// You can rate limit each user individually.
|
||
user: "jack",
|
||
// And have different rate limits for different tiers of users.
|
||
tier: "premium",
|
||
// You can also provide the tokens for each request.
|
||
// Tokens are flexible: LLM AI tokens in a prompt, complexity of a request,
|
||
// number of sub-actions, etc.
|
||
tokens: "200",
|
||
// When peak load exceeds external quotas or infrastructure capacity,
|
||
// requests can be throttled and prioritized.
|
||
priority: HIGH,
|
||
// Get deep insights into your workload. You can slice and dice performance
|
||
// metrics by any label.
|
||
workload: "/chat",
|
||
};
|
||
```
|
||
- **Wrap your workload** : Wrap your workload with`startFlow` and`endFlow` calls to establish control points around specific features or code sections
|
||
inside your application. For example, you can wrap your API endpoints with
|
||
Aperture SDKs to limit the number of requests per user or feature.
|
||
|
||
## Example
|
||
|
||
```
|
||
// Wrap your workload with startFlow and endFlow calls, passing in the
|
||
// labels you defined earlier.
|
||
const flow = await apertureClient.startFlow("your_workload", {
|
||
labels: labels,
|
||
// Lookup result cache key to retrieve a cached result.
|
||
resultCacheKey: queryParams,
|
||
});
|
||
// If rate or quota limit is not exceeded, the workload is executed.
|
||
if (flow.shouldRun()) {
|
||
// Return a cached result or execute the workload.
|
||
const cachedResult = flow.resultCache();
|
||
const result = await yourWorkload(cachedResult);
|
||
flow.setResultCache({
|
||
value: result,
|
||
ttl: { seconds: 86400, nanos: 0 },
|
||
});
|
||
}
|
||
//
|
||
```
|
||
- **Configure & monitor policies** : Configure policies to control the rate,
|
||
concurrency, and priority of requests.
|
||
|
||
## Policy YAML
|
||
|
||
```
|
||
blueprint: rate-limiting/base
|
||
uri: github.com/fluxninja/aperture/blueprints@latest
|
||
policy:
|
||
policy_name: rate_limit
|
||
rate_limiter:
|
||
bucket_capacity: 60
|
||
fill_amount: 60
|
||
parameters:
|
||
interval: 3600s
|
||
limit_by_label_key: user
|
||
selectors:
|
||
- control_point: your_workload
|
||
label_matcher:
|
||
match_list:
|
||
- key: tier
|
||
operator: In
|
||
values:
|
||
- premium
|
||
```
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
In addition to language SDKs, Aperture also integrates with existing control points such as API gateways, service meshes, and application middlewares.
|
||
|
||
- ⏱️
|
||
[**Global Rate and Concurrency Limiting**](https://docs.fluxninja.com/concepts/rate-limiter) :
|
||
Safeguard APIs and features against excessive usage with Aperture's
|
||
high-performance, distributed rate limiter. Identify individual users or
|
||
entities by fine-grained labels. Create precise rate limiters controlling
|
||
burst-capacity and fill-rate tailored to business-specific labels. Limit per
|
||
user or global concurrency of in-flight requests. Refer to the[Rate Limiting](https://docs.fluxninja.com/guides/per-user-rate-limiting) and[Concurrency Limiting](https://docs.fluxninja.com/guides/per-user-concurrency-limiting) guides for more details.
|
||
- 📊
|
||
[**API Quota Management**](https://docs.fluxninja.com/concepts/request-prioritization/quota-scheduler) :
|
||
Maintain compliance with external API quotas with a global token bucket and
|
||
smart request queuing. This feature regulates requests aimed at external
|
||
services, ensuring that the usage remains within prescribed rate limits and
|
||
avoids penalties or additional costs. Refer to the[API Quota Management](https://docs.fluxninja.com/guides/api-quota-management/) guide for more details.
|
||
- 🚦
|
||
[**Concurrency Control and Prioritization**](https://docs.fluxninja.com/concepts/request-prioritization/concurrency-scheduler) :
|
||
Safeguard against abrupt service overloads by limiting the number of
|
||
concurrent in-flight requests. Any requests beyond this limit are queued and
|
||
let in based on their priority as capacity becomes available. Refer to the[Concurrency Control and Prioritization](https://docs.fluxninja.com/development/guides/concurrency-control-and-prioritization/) guide for more details.
|
||
- 🎯
|
||
[**Workload Prioritization**](https://docs.fluxninja.com/concepts/scheduler/) :
|
||
Safeguard crucial user experience pathways and ensure prioritized access to
|
||
external APIs by strategically prioritizing workloads. With[weighted fair queuing](https://en.wikipedia.org/wiki/Weighted_fair_queueing) ,
|
||
Aperture aligns resource distribution with business value and urgency of
|
||
requests. Workload prioritization applies to API Quota Management and
|
||
Concurrency Control and Prioritization use cases.
|
||
- 💾 [**Caching**](https://docs.fluxninja.com/concepts/cache) : Boost application
|
||
performance and reduce costs by caching costly operations, preventing
|
||
duplicate requests to pay-per-use services, and easing the load on constrained
|
||
services. Refer to the[Caching](https://docs.fluxninja.com/guides/caching) guide for more details.
|
||
|
||
Note
|
||
|
||
FluxNinja has been acquired by [CodeRabbit](https://coderabbit.ai). New
|
||
sign-ups are temporarily disabled. Existing users can continue to use Aperture
|
||
Cloud by signing in to their accounts.
|
||
|
||
The easiest way to try Aperture is to
|
||
[sign up](https://app.fluxninja.com/sign-up) for a free Aperture Cloud account.
|
||
Aperture Cloud is a fully managed service by FluxNinja. With Aperture Cloud,
|
||
there's no need to manage any infrastructure, and you can integrate your
|
||
application with Aperture using SDKs. For more information, refer to the
|
||
[get started](https://docs.fluxninja.com/get-started/) guide.
|
||
|
||
|
||
|
||
|
||
|
||
*Prioritization Metrics for gpt-4*
|
||
|
||
|
||
|
||
|
||
*Performance Metrics for OpenAI Models*
|
||
|
||
To try Aperture in a local Kubernetes environment, refer to
|
||
[Playground docs](https://github.com/fluxninja/aperture/blob/main/playground/README.md).
|
||
|
||
- [Concepts](https://docs.fluxninja.com/concepts) section in Aperture
|
||
documentation provides an overview of Aperture's features and architecture.
|
||
- [Guides](https://docs.fluxninja.com/guides) section in Aperture documentation.
|
||
- [Observability-driven Load Management](https://docs.google.com/presentation/d/1Funig5u63w96G91ZjHV7ftgLOl1c5z5yDeuoVRH1QSs) deck covers the overview and use-cases of Aperture.
|
||
|
||
[Reporting bugs](https://github.com/fluxninja/aperture/issues/new?assignees=&labels=bug&template=bug_report.md&title=)
|
||
helps us improve Aperture to be more reliable and user-friendly. Include all the
|
||
required information to reproduce and understand the bug you are reporting.
|
||
Follow helper questions in the bug report template to make it easier. If you see
|
||
a way to improve Aperture, use the
|
||
[feature request](https://github.com/fluxninja/aperture/issues/new?assignees=&labels=feature+request&template=feature_request.md&title=)
|
||
template to create an issue.
|
||
|
||
To contribute code, read the [Contribution guide](https://github.com/fluxninja/aperture/blob/main/CONTRIBUTING.md).
|