Fault Handling with Polly – A Beginners Guide

Polly is the de-facto resilience and transient fault handling library for .NET. This article is a step by step beginners guide to using some of the most useful features of Polly.

What You’ll Learn

This step by step tutorial will teach you:

• What transient faults are
• The polices that can be used to handle transient faults
• How you can use Polly to implement those policies

What are transient faults?

Before answering that, let’s take a quick look at the definition of transient:

Transient: lasting only for a short amount of time; impermanent

Fairly straightforward then, transient faults relate to fault occurrences that exist for short periods of time. Examples of transient faults are:

• A server is refusing connections because it’s overloaded due to a demand spike
• A network connection is unavailable while a router reboots

So we’re really talking about conditions where it’s conceivable that if you retried the same request again, that it could succeed.

While I’m sure you can imagine many different types of transient fault, in this guide we’re going to turn our attention to a request service attempting to call a response service using HTTP, where the response service is issuing intermittent error responses.

Why do we care?

You may ask the question: if service is responding with an error, shouldn’t we just handle that error and move on? We could of course do that, and in the instance where the fault is “permanent”, (non-transient), e.g. you’re attempting to authenticate to an endpoint with incorrect credentials, that is basically what you would need to do.

In the case of transient faults though, we have opportunity to effect a successful end result by “trying again”. Indeed the rest of this article is concerned with how we can try again by implementing 1 or more resilience policies.

Finally, with the prevalence of distributed systems, (e.g. Microservices architectures), the need to ensure the resilience of those platforms should be front of mind. Especially when you consider, (given their distributed nature), their propensity to suffer from transient-type faults.

How can we deal with transient faults?

We could attempt to deal with transient faults by ourselves by writing loads of (probably error-prone), code, but why bother when we already have an awesome .NET library that’s done all the hard work for us? Introducing Polly:

The de-facto .NET resilience library

Polly gives us the ability to perform transient fault handling by employing the use of something it refers to as “Policies”. Now, while I don’t want to simply repeat the documentation already provided by the team at Polly, (which can be found here), it’s worth looking in a bit more detail at the Retry Policy, as that’s what we’ll be working with in this guide.

The Retry Policy

Probably the most obvious of all the policies – the retry: try the request again, (and again?) to see if it succeeds this time. What may be less obvious is that there are a number of things we can tweak when it comes to retrying like:

• Number of retries (we don’t necessarily want to retry forever)
• Time interval between retries (constant/linear or varying)

In this guide we’re only going to work with the retry policy, but we are going to employ 3 different approaches based on the time interval employed. Below is an illustration of a liner retry interval, (although we’ll also work with Immediate and Exponential Back-off intervals too). :

2 Retries with 10 seconds between each try (linear)

The “other” policies

As mentioned, Polly offers a range of other policies you can work with. To check out the full list take a look at the Official Polly Documentation. In getting started with the Retry Policy in this guide, you shouldn’t find it too difficult to employ the other polices if they suit your use-cases better.

Building Our Solution

Our solution will be comprised of 2 services:

• Request Service
• Response Service

We’ll create 2 separate .NET 5 projects for each, both of which will use the webapi template. We’ll start with our Response Service first which will act as our intermittently failing service, before moving on to the Request Service which will contain out Polly-centric code.

The code in this guide can be found on GitHub.

Tools and Environment

Before we get started it’s probably worth noting what you’ll need if you’d like to follow along.

.NET SDK

You’ll need version 5 of the framework: you can check to see if you have it installed by typing the following at a command prompt:

dotnet --version

You should get a response similar to the following:

If you get a lower version, or get an error of some kind, you can download the latest version of .NET here: https://dotnet.microsoft.com/download

Text Editor or IDE

I’m using VS Code, however this choice is up to you and shouldn’t make too much difference either way. If you want to checkout the excellent VS Code you can get it free here: https://code.visualstudio.com/download

Users of Visual Studio and other text editors should be able to follow along fine.

API Test Client

I’m going to use a tool called Insomnia to test out services, you can get it for free here: https://insomnia.rest/download

I like Insomnia for it’s clean interface and excellent GraphQL support (although we’re not doing anything with GraphQL here – I use GraphQL a lot so I love this tool).

If you’d rather use Postman, Fiddler, or I guess even a web browser you can do that, but I won’t be providing directions on how to use those.

Response Service Set Up

The Response Service will act as our endpoint that has intermittent issues, it’s really very simple comprising 1 Controller and 1 Action. There’s no model classes, database connections or anything like that, we’re using it simply as something to call.

At a command prompt, move into your working directory for the solution (we’ll create 2 separate projects here), and type:

dotnet new webapi -n ResponseService

This will create a new webapi template project called ResponseService, (with a folder of the same name). Open this folder with your editor of choice, if you’re using VS Code you can type:

code ResponseService

This should open the ResponseService project in a new VS Code window. The first thing we want to do is delete some of the template code files that we’re not going to use, so remove the following files highlighted below:

To be clear you should delete the following files, (and don’t delete any folders):

• WeatherForecast.cs
• WeatherForecasrController.cs

Once deleted, right click the Controllers folder, select New File and create a new file called: ResponseController.cs as shown below, (don’t forget the .cs extension):

Next, in the ResponseController.cs file, we’re going to create a new API Controller that’s going to host our endpoint, so enter the code as follows:

using Microsoft.AspNetCore.Mvc;

namespace ResponseService.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ResponseController : ControllerBase
    {
        public ActionResult GiveAResponse()
        {
            return Ok();
        }
    }
}

This isn’t really a tutorial on .NET APIs but if you’re interested in a fuller article explaining things in more detail, check out this article. I’ve highlighted some core concepts below:

1. This is specifying the base route for the controller – you’ll see this in action shortly
2. Class definition for our controller, note it inherits from ControllerBase
3. A very simple ActionResult that always returns success (HTTP 200)

Testing Our Response Service

Before we go too much further, let’s just check that everything we’ve done so far works, it also gives us an opportunity to fire up Insomnia and take it for a spin too.

So make sure you Save your work and at a command prompt, (make sure you are “in” the ResponseService folder), type the following:

dotnet run

You should see something similar to this:

Next start up Insomnia, right click in the left-most panel in your workspace and select New Request:

In the the New Request box, give your request a name (it can be anything but make it meaningful), and ensure that “GET” is selected:

Once you’re happy, click create, then add the following destination to the URL box:

The destination is made up of:

• The base server URL
• The base controller route, in this case: api/response

Click Send and you should get 200 OK response.

Making Our Service Fail

So as our service is always going to return a successful response, (which we don’t really want), we’re going to introduce some intermittent failing responses. We’ll do this by generating a (pseudo) random number between 1 and 100, and based on it’s value (e.g. greater than 50), we’ll issue a failing response, otherwise send an Ok. Using this method we can play with how likely the service is likely to fail, (or succeed depending on your point of view.)

With this in mind, update our controller action as shown below (the highlighted sections are the updates):

The controller action now expects an “id” as part of the route, this id is the lever we can pull to determine how successful our server is. We can test in Insomnia by updating our URL destination with a value (the id).

In the above example I supplied a value of 25, which means the server is more likely to return a HTTP 500 response (a failure), but of course it should be noted that we could of course get successes too.

With that, our Response Service is finished!

Request Service Set Up

Moving back to a command prompt and making sure you are in the root of your “solution” workspace, (tip: doing a directory listing you should see the ResponseService folder), we’re going to create our 2nd service as follows:

dotnet new webapi -n RequestService

Hitting “enter”, the 2nd project should be created, and again if you’re using VS Code you should be able to open the project in a new VS code window by typing:

code RequestService

As we did with the ResponseService, delete the following 2 files:

• WeatherForecast.cs
• WeatherForecastController.cs

Then create a file called RequestController.cs in the Controllers folder, when done your project structure should look like this:

Next we want to add the Polly library to our project, so making sure you are at a command prompt “inside” the RequestService folder, type the following, (tip: doing a directory listing you should see the RequestService.csproj file):

dotnet add package Microsoft.Extensions.Http.Polly

Note: this package includes the Polly package, but we’re making use of this as we’ll eventually be making use of a HttpClientFactory later in our build, which requires this particular package reference.

The RequestService.csproj file should look like this:

As we are going to have this service running at the same time as out ResponseService, we’ll also need to change the port that this service starts on. To do this, open the launchSettings.json file in the Properties folder and update the applicationUrl attribute in RequestService settings as shown below:

RequestService launchSettings.json file with Port allocations changed to 6000 and 6001

You can use any port allocation you like, but if you’re following along I’d suggest 6000 fro HTTP and 6001 for HTTPS to avoid confusion. Don’t forget to save and close the file when you’ve made the update.

Back to the Controller

Moving back to the RequestController.cs file, add the following code to start the build of our controller action.

using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;

namespace RequestService.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class RequestController : ControllerBase
    {
        public async Task<ActionResult> MakeRequest()
        {
            return Ok();
        }
    }
}

You’ll get a warning about the action result (MakeRequest) not containing an operator that uses the “await” keyword, ignore this for now as we’ll fix it below. Eventually we’ll add the code in here that makes a request to our ReponseService, for now I just want to get a very basic version of our controller action up and running so we can test. To do so (again make sure you save your code), type the following at a command prompt “inside” the RequestService folder:

dotnet run

You’ll get a warning about the lack of an await keyword, but you should also see that the service is running on Ports 6000 and 6001 as per the changes I made to the launchSettings.json file:

Moving back to Insomnia, create a 2nd request with the following attributes:

• Request Name: Indirect Request
• HTTP Method: GET
• Request URL: https://localhost:6001/api/request

Note: Pay particular attention to the Request URL, and make sure you’re using both the correct port and routing endpoint for our RequestService. Your set up in Insomnia should look like the following:

Click Send and you should get a HTTP 200 OK response. Now time to make a request to our Response Service…

Using HttpClient to make requests

We’ll start with some basic requests to our ResponseService using only a HttpClient, next we’ll introduce our Polly Retry policy and then finally we’ll use a HttpClientFactory to round out our build. To begin with though, update the RequestService controller action as follows, making sure you add the necessary using statements at the top of the file:

I’ve highlighted some points of note:

1. We create a HttpClient using the new keyword. Note this is not the recommended way of using HttpClient, but we’ll address this later, for now it’s fine…
2. Using our HttpClient we make an Asynchronous GET request to our ResponseService. The use of the await keyword here removes our warning, making out ActionResult truly asynchronous
3. We check the response of our request and depending on whether it’s OK or not return the appropriate response.

At this point in time, this code does exactly the same thing we have been doing with Insomnia. So to test it, save the changes, and if necessary re-start the RequestService (CTRL +C then dotnet run again), then make some calls to our RequestService from Insomnia once again. Looking at the console output for the RequestService you’ll see something like this:

Make sure you make a few requests and depending on what value you pass over to the ResponseService, the number of Failures Vs Successes should be as you expect.

At this point the RequestService is simply making requests to the ResponseService and passing back the response directly, it is not dealing with our (engineered) intermittent faults… Indeed if you look at the console output for the ResponseService, there will be a 1 to 1 mapping with the output of the RequestService.

Using Polly

Ok the moment we’ve been waiting for, we’re going to introduce a Polly retry policy to the mix. Now there are a number of ways you can “wire-up” polices, but I’m just going to use “Policy Class” and inject it into our RequestController using Dependency Injection.

In the RequestService, create a new folder in the root of our project called “Polices” and then add a file in that directory called “ClientPolicy.cs“, the results should look like this:

Now add the following code to the ClientPolicy.cs file:

using System.Net.Http;
using Polly;
using Polly.Retry;

namespace RequestService.Policies
{
    public class ClientPolicy
    {
        public AsyncRetryPolicy<HttpResponseMessage> ImmediateHttpRetry { get;}

        public ClientPolicy()
        {
            ImmediateHttpRetry = Policy.HandleResult<HttpResponseMessage>(
                res => !res.IsSuccessStatusCode)
                .RetryAsync(10);
        }
    }
}

I’ve provided some annotations on what’s going on below:

1. Public Retry property working with HttpResponseMessages failures
2. If our HttpResponse is not successful we retry 10 times

So basically this policy will immediately retry our request if it’s unsuccessful, and will attempt this 10 times. If we get a successful response at any time we’ll of course reply with that. If the 10th and final retry is a failure then we’ll return with that.

Make sure you save the file before moving on.

Using the Policy

First we need to register our newly create Policy Class in our DI container, so open the Startup.cs file in RequestService project and add the following to the ConfigureServices method:

services.AddSingleton<ClientPolicy>(new ClientPolicy());

In context this change should look like this, (make sure to include the necessary using statements at the top of the file):

Make sure you save the file before moving on.

The final step in this process is to inject and then use our ClientPolicy class in the controller, so back in the RequestController add the following code:

The code additions we made are as follows:

1. Added a reference to the ClientPolicy class
2. Created a class constructor so we can use constructor dependency injection
3. Created a private read only field so we can work with our injected class
4. Make use of the ImmediateRetryPolicy to execute requests to the ResponseService

You’ll note that I just commented out the original call so you can compare the differences, note that we’re still making use of our HttpClient.

Save the code and restart the RequestService as necessary and once again make a call using Insomnia, I got the following:

Now you’l more than likely get a different result from me, (indeed it’s possible you may still get a failure response), but what you will see is:

• A number of requests were made to the ResponseService
• Irrespective of the retry policy result, from the perspective of the RequestService we just have a single response (hopefully a success!).

To prove the point I made a number of further requests, and this was my result…

It just so happens that all my requests appeared successful (I’ve omitted the ResponseService logs, but there were of course many more failed requests than successful ones!). This is where you can now play around with the number of retries in the policy (e.g. reduce it to 3) and increase the likelihood of failure (or success) of the ResponseService.

So now you’re hopefully starting to see the power of Polly! But lets’ look at some other things we can do with Retry polices.

Give me a break!

In the above example, we just hammered the ResponseService. As soon as we got a failure, we just went immediately back and tried again. It may be more likely that in a genuine failure scenario, immediately retrying would be of little use, i.e. you’d probably just get another failure. A better approach may be to take some time between requests, if for no other reason than to give the failing service a break…

Back in our ClientPolicy class add a 2nd policy:

In this policy, I’ve changed the number of retries to 5, and I’m waiting 3 seconds between each failed attempt. To make sure we then use this Policy, back in the controller just update which Policy we want to make requests with:

Save everything and if necessary restart your RequestService, then make another request. Again you’re results will more then likely be different to mine, however this is what I experienced when I tried this policy:

1. I got 3 failed responses from the ResponseService with a 3 second wait between each
2. Total Request time was ~9s
3. I eventually got a Successful response.

You’ll note that this policy is probably not that great for “interactive” / real-time user-driven apps, but is probably ok for service to service communications where response times are not as important as eventual data consistency…

Before we move on to using a HttpClientFactory to create our HttpClient(s), (instead of “newing” them up), let’s take a look at 1 more retry policy: Retry with Exponential Back-off. This is really just a fancy way of saying that we’ll introduce a wait with each failed request, with the wait period increasing exponentially, i.e. initially we’ll retry quickly, but each successive retry will have a longer and longer wait.

Again the idea behind this policy is if the first few calls are failures, then we should probably wait increasingly longer between calls to give the failing service the chance to recover.

Back in our ClientPolicy class, add our 3rd and final policy:

Also don’t forget to update the controller to make use of this policy:

Save everything and restart your Request Service as necessary. I’ll let you test this policy yourself and observe the results. It should be quite obvious that the time between each retry gets longer…

Using HttpClientFactory

I’ve included this last section as it’s really best practice to use a HttpClientFactory to create your HttpClients. A HttpClientFactory manages the HttpClient connection pool for you, so it helps to avoids things like socket exhaustion. There is a detailed article from Microsoft on this subject that you can read at your leisure.

Now I’m going to take the most basic approach to using HttpClientFactory in this example, but you can do a lot more with it. To really make further use of it  in the context of Polly, (which we’re not covering today), you’d need to make use of either Named or Typed clients, along with defining your Polices in slightly different way.

The team at Polly have already documented this at length, so I don’t feel there’s much for me to add. So with that, let’s implement Polly with the HttpClientFactory.

Over in the RequestService Startup class add the following service registration in the ConfigureServices method:

services.AddHttpClient();

In context this change should look as follows:

The over in the RequestController, add the following updates:

1. Create a private readonly field for our HttpClientFactory
2. Inject an instance of HttpClientFactory in via constructor dependency injection
3. Assign out private instance of HttpClientFactory
4. Use the HttpClientFactory to create a HttpClient

As you can see the rest of the code remains unchanged.

Save your code and make some new requests, everything looks the same as before except we’re now getting our HttpClients via a Factory.

Next Steps

This was just an introductory “on-ramp” to Polly, as I actually couldn’t find something for a complete beginner – so hopefully you found it useful!

Polly is so much more powerful that I’ve outlined here, and can work with many different “fault-types” not just Http Messaging, as well as other polices. Hopefully you can take what you’ve learned here as a starting foundation to explore those more advanced concepts.