When Non-Prod Gets the Contract Change Before Prod

Non-prod is red and nobody touched anything.

Say you come in and non-prod is failing, and your code hasn’t changed since the last green run. The upstream API is labeled production-ready. Prod is healthy on your dashboards. A quick request comes back with an error saying a parameter is invalid. You dig into the provider’s code and find it. A parameter you send was renamed in non-prod, and their prod deploy is a couple of weeks out.

So you can’t just update. The new name only exists in non-prod, and prod still expects the old one. That means there are two contracts for the same API. Prod is what you ship against, and non-prod is what you test against. If you move to the non-prod contract, prod breaks on your next deploy. If you stay on the prod contract, non-prod stays red until their deploy. Maybe the feature isn’t critical and the impact is small. But a red environment that everyone learns to ignore is how real breaks slip through.

Production-ready is a product label. It means the endpoint is up, supported, and safe for traffic. It says nothing about how changes reach the teams that call it. Product says it’s ready and engineering keeps building it, and both are right. The provider treats non-prod as staging for their next release. Consumers treat non-prod as a preview of what they’re about to ship into. Nobody defined which one non-prod is, and that’s the real conflict. The upstream team didn’t do anything wrong by their own standards. What’s missing is an agreement.

Ask the provider for backwards compatibility and you’ll often get a yes with no date. A promise without a date can’t be planned around. What a consumer needs is “the old shape stays until this date.” Here’s how to set that up, from both sides.

Backwards Compatible by Default

So the default rule is backwards compatibility. Every change ships as an addition. A change that would break a caller either goes through expand and contract or gets a new version.

What counts as breaking

Breaking: removing or renaming a field, changing a type, making an optional field required, tightening validation, changing error codes or status codes, or changing default behavior.

Usually safe: adding an optional field, adding a new endpoint, or accepting a new optional parameter.

A renamed field is a breaking change. It’s the same as deleting the old field and adding a new one, and every caller still sending the old name fails. Some changes look additive and still break people. Adding a value to an enum breaks any client that switches on every value. Hyrum’s Law covers the rest: “With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.” Field order, error message text, and response timing all end up in someone’s code. So treat anything observable as part of the contract unless you’ve said otherwise in writing.

Expand and contract, also called parallel change, is how most breaking changes should ship. Add the new field next to the old one and accept both. Move consumers over, watch traffic on the old field until it hits zero, then remove it. Both shapes are live at the same time, so non-prod and prod never disagree. For the rename in this scenario, the provider would accept both parameter names until every caller had moved. I do this all the time on my own API and the web app that depends on it. When you own both sides, it’s pretty much an afternoon of work. When the consumer is another team, the old shape has to stay until they’ve moved, and that’s what needs an agreement.

I’d save new versions for changes that are too big to run side by side. Pick one versioning scheme and write down what a bump means. The common options are a major version in the path (/v1, /v2), a version header, or date-based versions pinned per caller. Stripe pins each account to a dated version, and GitHub uses an X-GitHub-Api-Version header with a date. For an internal API with a handful of consumers, I’d use the path. It shows up in every log line and every request, and nobody has to learn a header. Within a version, changes are additive only.

Deprecation needs to be something your code can see. Announcements in Slack or some channel are easy to miss, and I don’t always remember to check those channels. HTTP has standard headers for this. The Deprecation header (RFC 9745) marks a resource as deprecated and says since when. The Sunset header (RFC 8594) says when it goes away. Pair them with a changelog and a fixed window. GitHub supports the previous REST version for at least 24 months, and Kubernetes keeps deprecated GA APIs for 12 months or three releases, whichever is longer. Internal APIs don’t need that long. A couple of consumer release cycles is usually enough. What matters is that someone writes the number down.

But compatibility runs both ways. Consumers have to stop breaking on things that don’t matter. Martin Fowler calls this the Tolerant Reader. Read only the fields you use, ignore unknown fields, and give unknown enum values a default branch. Validate strictly at your boundary, but only on what you depend on. A client that’s too strict turns every harmless addition into an outage. A client that’s too loose hides real breaks. Tolerant reading also only helps with responses. The break in this scenario is a renamed request parameter, and nothing on the consumer’s side absorbs that.

It also matters where your tests live. If your tests run end-to-end, from the frontend through your API to the upstream one, a failure doesn’t tell you whose problem it is. Where a test lives decides what its failure tells you. So put contract checks in your backend’s client for the upstream API. Live checks call non-prod and prod directly on a schedule, validate responses against your schemas, and alert on drift without blocking your builds. Stubbed tests replay recorded prod responses in CI, so they only fail when your code changes. Keep end-to-end tests for a few critical user journeys.

And it’s even better to catch drift before any test runs. If the provider publishes OpenAPI specs, pull the non-prod and prod specs on a schedule and diff them with oasdiff. The difference between those two specs is exactly what’s about to reach prod. A breaking diff opens a ticket while there’s still time to act, and non-prod never goes red.

Consumer-driven contract tests go further. With Pact, you publish the requests you make and the responses you rely on, and the provider’s CI verifies them before they deploy with can-i-deploy. The rename would have failed their pipeline before it ever reached your non-prod environment. It needs the provider to opt in, but it doesn’t need an OpenAPI spec. Plenty of internal APIs don’t have one. It’s just requests and code. In that case the consumer writes the contract. Pact tests, runtime schemas like Zod for the fields you use, and saved prod responses to compare non-prod against all work. And the cheapest thing you can ask a provider for is a spec generated from their code.

And consumer-written contracts turn out to be the stronger design for internal APIs anyway. A provider spec describes everything the API offers. Consumer contracts describe what callers actually use, and that’s what a provider needs to know before changing anything.

Negotiating the Agreement

The technical pieces are the easy part. Getting another team to agree to them is harder. A good agreement covers five things. The spec lives in a repo and is versioned. Breaking changes go through expand and contract or a version bump, with a notice period and a sunset date. Non-prod keeps the current prod contract reachable. Every change gets a changelog entry. And every API has a named owner and a list of consumers. If you can only get one, get the rule on breaking changes. Everything else depends on it.

Non-prod should show what prod is about to be. The provider can do that a few ways. They can run the new version next to the old one. They can let callers opt in to new behavior with a header or flag. Or they can do unreleased work in a separate environment. Lead with whichever costs the upstream team the least.

When you ask, bring evidence and propose something small. Count the non-prod failures caused by upstream changes and the hours spent tracing them. Your boundary checks already log that. Show what the provider gets out of it: fewer “why is non-prod broken” threads and fewer interruptions from consumers. Start with one rule. The current prod contract stays reachable in non-prod, and new shapes are added next to it. Add more once that one holds.

So if you’re consuming an API that keeps changing in non-prod, here’s the order I’d take it in:

1. Make the current gap explicit. Either track the red non-prod environment as a known issue that closes on the prod deploy date, or add a small environment-specific mapping in your client with a date to remove it.
2. Make your client tolerant.
3. Add live boundary checks.
4. Diff the non-prod and prod specs, if there are specs.
5. Log Deprecation and Sunset headers.
6. Track how often upstream changes break non-prod.
7. Propose a short agreement, starting with keeping the current contract reachable in non-prod.
8. Add contract tests once there’s trust.

The first six don’t need anyone’s permission.

All of this assumes you know who to talk to. Turn it around and look at the provider. Every provider has consumers it doesn’t know about. In this scenario, the provider almost certainly didn’t know you sent that parameter. In a small organization, a conversation fixes that. Past a certain size, one-on-one communication is the bottleneck, and the agreement has to be automated.

Here’s how I think about scaling it out. Every team publishes its contract to a central registry as part of CI, and the registry tracks which version runs in each environment. Every caller identifies itself with a client ID, so traffic shows who calls what, down to the field. Consumer contracts fill in what traffic can’t see. On every change, a checker diffs the contract and labels it additive, breaking, or a deprecation. A breaking change without expand and contract or a version bump fails the provider’s build. Only the teams that use the changed field get a ticket, with the date the change reaches prod. In this scenario, that ticket would have shown up before non-prod went red.

Pieces that already exist

Schema registries like Confluent Schema Registry reject event schema changes that would break readers.

Pact Broker stores consumer contracts centrally, runs can-i-deploy checks per environment, and sends webhooks when a contract changes.

Service catalogs like Backstage track which services provide and consume which APIs, and which team owns each one.

There are a couple of hard parts. If every additive change pings everyone, people stop reading, so alerts have to target the field level. And a schema can’t see a change in meaning, like a field switching from cents to dollars. Those still need a person and a changelog entry. Someone also has to own the registry, which makes it one more internal API with consumers of its own. The rules are the same ones from the two-team agreement. The registry enforces them for every team at once.

An API contract is a social agreement first. The spec just writes it down.

Cheers,

Will