Resilience testing
Every API a program depends on will, one day, be slow, wrong or gone. Resilience testing asks what your side does then. Does the app show a clear message and recover, or does it freeze, lose the user's work, or charge a card twice?
These failures are hard to produce with a real service and easy to produce with a mock. That is the method: point the client at Mock Studio, make the mock misbehave in one specific way, and watch.
The failures worth testing
| Failure | What the client sees | Set up in Mock Studio with |
|---|---|---|
| Slow response | A long wait, then a normal answer | Latency on the endpoint or response |
| Variable speed | Sometimes fast, sometimes slow | Latency plus jitter |
| Timeout | No answer before the client gives up | Hang (never respond), or latency longer than the client's timeout |
| Server error | 500, 502, 503 or 504 | A response with that status |
| Occasional error | Mostly fine, sometimes a 5xx | Random / weighted |
| Fails, then recovers | Error, error, success | Sequence (round-robin) |
| Rate limiting | 429, usually with Retry-After | A response with status 429 and that header |
| Connection dropped | No status at all: a network error | Drop connection |
| Empty body | A success status with nothing in it | Empty response |
| Corrupt body | A success status with text that does not parse | Malformed body |
| Wrong shape | Valid JSON, missing a field the client needs | A response with the field removed |
| Expired credentials | 401 in the middle of a session | A rule, or a sequence that turns to 401 |
The Flaky Payments sample has most of these ready to run. Latency and faults describes every setting.
What good behaviour looks like
For each failure, decide what should happen before you test. Typical expectations:
| When | The client should |
|---|---|
| A response is slow | Show that it is working, and stay usable |
| A request times out | Stop waiting after a set time and say so |
| A 5xx arrives on a read | Retry a few times with growing pauses, then report |
| A 5xx arrives on a write | Retry only if the request is safe to repeat |
| A 429 arrives | Wait as long as Retry-After says, then retry |
| The body cannot be parsed | Treat it as a failure, not as empty data |
| A 401 arrives mid-session | Refresh the token or ask the user to sign in again |
| Anything fails | Keep what the user typed, and never show a raw error dump |
The method
Pick one failure. One at a time, or you cannot tell which one caused what you see.
Configure the mock. Change one endpoint to produce that failure.
Exercise the client. Use the app, run the script, or send the request from ApiWorkbench if the "client" is your test suite.
Observe three things. What the user sees, what the client sends next (in Mock Studio's Live log), and whether everything works again afterwards.
Restore the endpoint, and confirm the client recovers without a restart.
Step 4's second point is the one people skip. The Live log shows whether the client retried, how many times and how far apart. A client that retries fifty times in a second is making the outage worse.
Retries
Retrying is the commonest defence and the easiest to get wrong. Test these cases with a Sequence:
| Sequence | A good client |
|---|---|
| 503, 200 | Succeeds on the second try, and the user never knows |
| 503, 503, 503, 503, 503… | Gives up after a few tries and reports |
429 with Retry-After: 2, then 200 | Waits about two seconds before the retry |
| 400, 200 | Does not retry. A 4xx will fail the same way again |
Check the pauses in the Live log. Each wait should be longer than the last (backoff), and not all clients should retry at the same instant.
Retrying writes
A POST that times out may still have happened on the server. Retrying it can create a duplicate order or a second charge. The fix is an idempotency key: a unique id sent with the request so the server can recognise a repeat. To test it, send the same request twice with the same key and confirm only one thing was created.
Timeouts
A client with no timeout waits for ever. Test with Hang (never respond):
- Does the client give up? After how long?
- Is that wait reasonable for a person looking at a screen?
- Does it free its resources, or does every hung call pile up?
In ApiWorkbench, the connect and response timeouts are in Settings, and a request can have its own. See App settings and Per-request settings.
Slow is a failure too
Set a latency of a few seconds and use the app as a person would. Look for a screen with no sign of progress, a button that can be pressed twice (sending the request twice), and results that arrive out of order.
In a test suite, a Time below (ms) test turns "too slow" into a failure you can see. See Load testing for measuring speed properly.
Testing the API's own resilience
Everything above tests a client. To test how a server copes with its failing dependency, put Mock Studio in that dependency's place: the payment provider, the address lookup, the email service. Make the mock slow or failing, and then test your API from ApiWorkbench:
- Does your API return a clean 502 or 503 with a useful message, or a 500 with a stack trace?
- Does it answer in reasonable time, or hang as long as its dependency does?
- Does it recover by itself when the dependency comes back?
A resilience checklist
- [ ] Every call has a timeout
- [ ] A slow response does not freeze the interface
- [ ] 5xx on reads is retried, with backoff and a limit
- [ ] Writes are not retried blindly
- [ ] 429 is honoured, including
Retry-After - [ ] A dropped connection produces a clear message
- [ ] An empty or malformed body is treated as an error
- [ ] A 401 mid-session leads to a refresh or a sign-in
- [ ] The client recovers when the service does, without a restart
- [ ] No failure loses what the user entered