Microservices Behind an API Gateway: A Real Case Study

The first version of the library system was one Django app and it worked fine. It is still the thing I would build again for the same requirements. What changed was not the load — it was that four different teams needed to ship on four different schedules, and every deploy meant coordinating all of them.

That is the honest reason we split it, and it is worth saying plainly because "we needed to scale" is the reason people give and it is almost never true. This post is the architecture we ended up with: five services behind one gateway, what each boundary cost, the two boundaries I drew wrong, and the failure modes that only appear once a call can cross a network.

The system

Five services and why each one exists

A public library: catalogue, borrowing, members, notifications, search. Roughly 40,000 members across a dozen branches — not a scale story.

The mistakes

Two boundaries I drew wrong

Both had the same shape and it took an incident each to see it.

The second was Holds, split out for exactly the same reason and pulled back for exactly the same one. The rule that came out of both:

The gateway

What belongs in it, and what must never

The gateway is the most tempting place to put things and the worst place to put most of them. Everything routes through it, so anything you add there becomes a dependency of the whole system.

Contracts

The interface is the product

Inside a monolith a bad function signature is a refactor. Across a network it is a coordinated release. Three rules kept that manageable:

1. Additive changes only, until a major version

   New optional fields are free. Removing a field, renaming one, or narrowing a type is a breaking change and needs a version. Consumers must ignore fields they do not recognise — write that down, because a strict parser on the consumer side turns every additive change into a breaking one.

2. The contract lives in the repo that serves it

   An OpenAPI spec next to the code, generated in CI, published as an artifact. A spec maintained separately from the implementation is documentation, and documentation drifts.

3. Consumer-driven contract tests in CI

   Lending's test suite asserts the shape it needs from Members. That test runs in Members' pipeline. This is the single highest-value thing on the list — it is the difference between finding a break at merge and finding it in production.

Failure

The modes that only exist once there is a network

This is the part the diagrams never show and the part that decides whether the split was worth it.

The timeout budget is the load-bearing idea

Every call in a chain having its own 30-second default is how one slow service takes down a system. The budget is set once at the edge and spent going down:

# The gateway gives the whole request 3 seconds and says so in a header.
# Each hop spends from what is left and passes the remainder on.

def forward(request, service):
    remaining = float(request.headers.get("X-Deadline-Ms", 3000))
    if remaining <= 50:
        raise DeadlineExceeded()          # do not start work you cannot finish

    started = time.monotonic()
    response = http.post(
        service.url,
        timeout=remaining / 1000,
        headers={"X-Deadline-Ms": str(remaining - 50)},   # reserve overhead
    )
    spent = (time.monotonic() - started) * 1000
    metrics.observe("hop_ms", spent, service=service.name)
    return response

The remaining <= 50 check matters more than it looks. Without it, a service at the end of a chain starts a two-second database query on behalf of a caller that gave up a second ago — burning capacity for a response nobody will read, at exactly the moment the system is already struggling.

Honesty

What this cost, and what should have stayed together

• Local development got much worse. Five services, a bus and a gateway is a compose file and a lot of RAM. New developers lose a day on it. This is a real, recurring tax that no diagram shows.
• Every cross-service change is two pull requests in an order that matters. Additive-first discipline makes it survivable rather than pleasant.
• Debugging needs distributed tracing from day one. Without a request ID threaded end to end, "it was slow" is unanswerable. We added it late and regretted the gap.
• Notifications did not need to be a service. It needed to be a queue. It is separate now largely because it was separate then, which is not a reason.
• If the team were still four people, one Django app would win. The split bought independent deploys for independent teams. That is the only thing it bought, and it is worth having only when you have the second thing.

The checklist

• Split for release cadence and team ownership, not for load you do not have.
• Never split a transaction — things consistent within one user action live together.
• The gateway does authentication, not authorisation, and holds no database.
• Contracts live with the code that serves them, generated in CI.
• Consumer-driven contract tests run in the provider's pipeline.
• A timeout budget from the edge, divided down the chain, checked before starting work.
• Retry only idempotent reads, with jitter and a circuit breaker.
• Anything that can be async, is — search projections and notifications never block a user.
• A request ID threaded through everything from the first day.
• Write down what each service owns, in one sentence. If you cannot, the boundary is wrong.

Where to start on Monday

Do not start by drawing services. Start by writing the one sentence each candidate service owns, then look for a user action that crosses two of them and has to be consistent. Every one of those is a boundary you are about to draw in the wrong place.

• service boundaries
• API gateway
• timeout budget
• circuit breaker
• contract tests
• event bus
• read projections
• request IDs

The real-time layer sits awkwardly on top of all this, which is its own problem and its own post: adding WebSockets to a microservice architecture. If you are running this on your own hardware rather than a managed platform, the CI/CD guide covers getting five services deployed without a cloud platform underneath you.