Data Contracts End the Blame Game

A familiar Monday: an executive dashboard shows revenue down sharply. Nothing has happened to revenue. A backend team renamed a field on Friday as part of a routine refactor, and a pipeline that depended on it has been silently producing nulls ever since.

The data team gets the escalation. But they could not have prevented this, because they had no way of knowing the change was coming. The dependency existed, and it was invisible to the only people who could have acted on it.

The structural problem

In most organisations, application databases became the analytics interface by accident. Nobody decided that the users table was a public API; a pipeline simply started reading it, and thereafter every schema change was a potential breaking change to a consumer the producing team did not know existed.

This is an interface without a contract. We would not accept it between two services. We version APIs, publish schemas, and treat a breaking change as an event requiring coordination. Analytics dependencies deserve the same treatment and rarely receive it.

What a contract actually contains

A data contract is a declaration by the producing team of what they guarantee. It is deliberately modest in scope. The value is in it existing and being enforced, not in its sophistication.

# contracts/orders.v1.yaml
dataset: orders
owner: checkout-team
version: 1

schema:
  order_id:    { type: string, required: true, unique: true }
  customer_id: { type: string, required: true }
  total_cents: { type: integer, required: true, min: 0 }
  currency:    { type: string, required: true, enum: [USD, EUR, GBP, INR] }
  placed_at:   { type: timestamp, required: true }
  status:      { type: string, required: true, enum: [pending, paid, shipped, cancelled] }

guarantees:
  freshness: 15m       # rows land within 15 minutes of the event
  completeness: 99.9%  # of orders present in the source system

evolution:
  additive_changes: allowed      # new optional columns ship freely
  breaking_changes: major_bump   # renames and removals require v2
  deprecation_notice: 90d

The evolution block is the part that changes behaviour. It states plainly that adding a field is routine and removing one is a coordinated event with a notice period. That single distinction resolves most of the day-to-day friction between producing and consuming teams.

Enforce in CI, where it is cheap

A contract that is only a document will drift within a quarter. Enforcement has to sit in the producing team's pipeline, so that a violation is caught by the engineer who introduced it, at the moment they introduce it.

• Validate the schema against the contract on every pull request. A removed field fails the build with a message naming the contract and its owner.
• Test the guarantees in production continuously. Freshness and completeness are runtime properties; assert them on a schedule and alert the producing team, not the consuming one.
• Publish the consumer list. When an engineer sees that four teams depend on this dataset, coordination stops being an abstract obligation.

Introducing contracts without stalling delivery

Attempting to contract every dataset at once fails predictably. It is a large amount of work with no visible benefit until the end, and it positions the data team as an obstacle.

1. Start with the datasets that have already broken. There is an incident to point at, so the value is not hypothetical and the producing team is already receptive.
2. *Write the first contracts for the producing teams, not with them.* Reduce the initial cost to reviewing a pull request. Ownership can transfer once the value is evident.
3. Add enforcement only after the contract reflects reality. A contract that fails the build on day one because it describes an aspiration will be disabled within a week and never re-enabled.
4. Let demand pull the rest. Once one team stops being paged for upstream breakage, others ask for the same treatment. That is a far better adoption mechanism than a mandate.

The cultural shift matters more than the tooling. Contracts move the conversation from who broke the dashboard to what did we agree to, and that is a conversation both teams can have productively.