Circuit Breakers to handle failures in serverless applications

Introduction

When you are developing micro-services, you might find that there might be some dependencies which could fail or be temporary unavailable. Examples of these dependencies are calls to other services, data stores, etc. These dependencies might be unavailable for multiple reasons such as maintenance, throttling, etc. 

The idea behind the circuit breaker pattern is wrapping the dependency, so if there is a consistent failure, we temporary stop trying to call the dependency and instead we can have define an alternative fallback function.

Normally a circuit breaker can be in three different states:

  • Closed: we call the dependency as usual. If there are failures, we track them and we open the circuit if a defined threshold is reached.
  • Open: instead of calling the dependency, we call the fallback function. If we don't define a fallback function, it will fail early, without calling the dependency. This can prevent overloading an unhealthy dependent service or data store. While the circuit breaker is OPEN, if we reach a timeout, we will change the status to HALF.
  • Half (open): if the request is successful, the status will change to CLOSED and it will start processing all incoming requests. However, if the requests are still failing, the CircuitBreaker will change to OPEN.
The following diagram illustrates the different statuses and transitions:

Proposed Architecture for serverless applications

The Circuit Breaker pattern is a well know pattern and some popular libraries such us Polly (https://github.com/App-vNext/Polly) implement this pattern. In the implementation offered by Polly, the status of the circuit breaker is persisted in memory. This works well for single instance services, but if we are running multiple instances of the service concurrently, you need to share the status of the circuit breaker, so all the instances are consistently handling the circuit breaker. This applies for AWS lambda functions or ECS if we are running multiple nodes in parallel. For lambdas, you also have to bear in mind that if the lambda gets cold, it will erase the memory. For that reason it is important to persist the status of the circuit breaker.

Regarding implementations more focus on serverless, I found this implementation written in Node.js. Since our micro-services are implanted using .NET 6, I couldn't use a node.js implementation. In addition, that implementation is meant to work at lambda level and for my scenario I wanted something more generic, which can be used by any service and any dependency.

Regarding the persistence, in order to have lower latency reading/updating the state, we decided to store the status of the circuit breaker in Amazon MemoryDB, which is based on Redis with persistent storage. The object we store to persist the status is really small and we want to make the storage as performant as possible. That is the reason why Amazon MemoryDB can be a better candidate than DynamoDB for storing circuit breaker objects.

Based on those principles, the proposed architecture is displayed in the following diagram, where a lambda might have two circuit breaker to manage two dependencies and persist the status in Amazon MemoryDB. In the diagram we see a lambda which has two dependencies, Amazon ElasticSearch and a external API, so we can define one circuit breaker for each one. In addition, we can also define a fallback function for each dependency, the failure handling process could be different depending on the dependency.

.NET 6 implementation

You can find the full source code of this implementation on my GitHub repository: https://github.com/albertocorrales/circuitbreaker-dotnet.


First of all, you need to configure the dependency injection

internal static void RegisterCircuitBreakers(this IServiceCollection services, IConfiguration configuration)
{
    services.Configure<ElastiCacheConfig>(configuration.GetSection("CircuitBreaker"));
    services.AddSingleton<ICircuitBreakerFactory, CircuitBreakerFactory>();
    services.AddSingleton<ICircuitBreakerRepository, CircuitBreakerRepositoryElastiCache>();
}
For the persistence, we have a generic repository defined by the interface ICircuitBreakerRepository. Then we could implement other persistence storages if we want to. For this implementation, as it was discussed above, we are using MemoryDB, so the implementation of the repository is provided by the class CircuitBreakerRepositoryElastiCache. 

In addition, we are injecting the class CircuitBreakerFactory, so we just need to have this dependency in the classes where we are using circuit breakers.

Then you can use it in your Web APIs, Lambdas, etc. For example in a Web API controller, we can it as follows:
public class MyController : ControllerBase
{
    private readonly ICircuitBreakerFactory _circuitBreakerFactory;

    public MyController (ICircuitBreakerFactory circuitBreakerFactory)
    {
        _circuitBreakerFactory = circuitBreakerFactory;
    }

    public async Task<IActionResult> MyMethod([FromBody] RequestModel request)
    {
        var circuitBreakerOptions = new CircuitBreakerOptions<ResultModel>("circuit-breaker-id-123", () =>  asyncFunction(request));
        var circuitBreaker = _circuitBreakerFactory.CreateCircuitBreaker<ResultModel>(circuitBreakerOptions);

        return await circuitBreaker.Fire();
    }
}

In the model CircuitBreakerOptions, we can configure the following properties: 
  • CircuitBreakerId (required): id for the circuit breaker. This value has to be unique, but we could also use the same value if for example we want to use the same circuit breaker in two places. (For example, if EventStore is down, it might affect projections and commands).
  • Request (required): function to wrap under the circuit breaker.
  • FailureThreshold: number of failure executions that should happen until the circuit is OPEN.
  • SuccessThreshold: number of successful executions to change the status of the circuit breaker to CLOSED.
  • Timeout: number of milliseconds to try a new request since the circuit breaker changed to OPEN.
  • Fallback: alternative function to execute if the main function fails or the circuit breaker is OPEN.

On the other hand, the CircuitBreakerModel that we persists, it is defined by the following properties:
  • Id: unique id to identify the circuit breaker
  • Status: current status of the circuit breaker (open, closed or half)
  • FailureCount: number of consecutive failure requests
  • SuccessCount: number of consecutive successful requests
  • NextAttempt. datetime when the circuit breaker is closed and we should try next time.

Conclusions

In this article we have seen the importance of managing failures properly in your applications with circuit breakers. In addition, we discussed about the reason why multi-node applications should persist the status of the circuit breakers, so all the instances can work consistently. 

Finally, we have seen an implementation for .NET 6, which helps us to manage and persist our circuit breakers. The implementation is flexible and it allows us to define multiple circuit breakers and handle them independently. Furthermore, we can use MemoryDB as persistent storage or implement our own repository.

Comments

Post a Comment

Popular posts from this blog

Building Micro-Frontends with Single-Spa

Image
Introduction Micro-services architecture is becoming trendy since it is a suitable option many projects, specially those that are complex and we want to be able spilt our business logic into different services that can be built and deliver independently. In spite of the fact that micro-services architectures are more complex, they increase the resilience of your system and increase the time-to-market delivery, which can be crucial for many companies. But it is not all about services and backend. With the introduction of single page applications (SPAs), we can build fully-functional user interfaces that can work independently in the browser. And here we should ask ourselves the same questions as for backend applications: Is my SPA becoming a huge monolith? Would I like to deliver functional modules independently? If you using micro-services, is there a relation between my services and your UI? Do we want to have functional teams that can deliver new features (frontend+backend) independe...
Read more

AWS assuming role + MFA with pulumi (and other applications)

Image
Introduction In AWS, managing your IAM in a centralized account is a good practice, so then you can assume a role to access to other accounts without need of duplicating users. You can find more info about this approach in this article . In addition, for security reasons, it is highly recommended to enable MFA (Multi-factor Authentication), so you need a device to generate a temporary code to get access. In the image below it is showed how assume role with MFA works, but you also have an article about MFA in this link . This approach is really useful and most of the configuration can be automated with pulumi (or other providers like terraform or CloudFormation). And once it is configured, you can switch your role and get access to other accounts with a few clicks.  However, when it comes to external software such us pulumi, getting access assuming a role and using MFA might not be too straightforward. In this article we will see how this can be configured and how we can easily refr...
Read more

Managing snapshots for Amazon ElasticSearch with Dotnet Core Lambdas

Image
Introduction It is awesome to have some useful services like ElasticSearch managed by AWS , so you don't have to care about patching, monitoring, etc.  When it comes to backup management for indexes, Amazon ElasticSearch includes automated snapshots. Automated snapshots are only for cluster recovery. You can use them to restore your domain in the event of red cluster status or other data loss. Amazon ES stores automated snapshots in a preconfigured Amazon S3 bucket at no additional charge. In particular, if you are using a higher version than 5.3, Amazon ElasticSearch takes hourly automated snapshots and retains up to 336 of them for 14 days. However, this policy might not be enough if you need to keep your snapshots for a longer 15 days, or you need to take a snapshot to create a new cluster in another region, subnet, etc. If you have different requirements to manage your snapshots, you will have to work with manual snapshots. For this part, AWS does not offer a good managed exper...
Read more