Schema normalisation
Can N heterogeneous source schemas be collapsed into one canonical model such that adding source N+1 touches exactly one file — and nothing downstream learns that the source exists?
- success
- a new adapter ships without editing the scorer, the store, or the view
- failure
- any source-specific branch appears outside its own adapter
- also testing
- whether an undocumented upstream field rename is caught at the boundary
Five exchanges, five answers to the same question
CryptoFundingWatch exists to answer one question: which perpetual has the best funding spread right now? Answering it means putting five venues’ numbers side by side. The venues make that surprisingly hard.
They disagree on the identifier — the same instrument is BTCUSDT, BTC-USD-SWAP, BTC, BTC-PERP, and PF_XBTUSD depending on who you ask. Kraken still calls Bitcoin XBT.
They disagree on the field name and type — lastFundingRate as a string, fundingRate as a string, funding as a string, funding_rate as a string, fundingRate as a float.
They disagree on the timestamp — epoch milliseconds, epoch milliseconds as a string, epoch seconds, ISO 8601, ISO 8601 with milliseconds.
And most dangerously, they disagree on the funding interval. A rate quoted per eight hours and a rate quoted per hour are not the same kind of number, and nothing in the payload shouts about it. Put them in the same column and the ranking is simply wrong — quietly, plausibly, and in the direction that loses money.
LatticePulse has the harder version of this problem. Its four sources aren’t different formats of the same thing; they’re different kinds of thing. A Reddit thread with 284 upvotes and an insider buying 12,000 shares are both evidence about a ticker, but there is no honest exchange rate between them. Forcing both into one number at the adapter throws away the distinction the scorer needs.
Adapters own every difference
One adapter per source, each a pure function from that source’s raw payload to the canonical model. Everything downstream is source-agnostic by construction — not by discipline.
Run the adapters
Representative payload shapes for each source. Press run and watch them collapse into one model — then break one and see what the validator does.
0.00010000 and Hyperliquid quotes 0.0000125 — an eight-fold difference in the raw payload. Annualised, they are the same number. That is the entire argument for normalising at the boundary.| press “Run adapters” |
Six steps, in this order
The order matters more than any individual step — defining the canonical model after writing adapters produces a model shaped like whichever source came first.
Define the canonical model from the consumer, not the sources
Ask what the scorer and the view need, and design backwards. A model built as the union of every source’s fields inherits every source’s accidents — and grows a nullable column each time you add a venue.
One adapter per source, as a pure function
Raw payload in, canonical records out, no I/O and no shared state. Pure means the adapter is testable against a saved fixture without a network, which is what makes the conformance suite in step six possible.
Validate on arrival — reject rather than coerce
A missing or unparseable required field marks that source degraded. It never emits a row with a null rate, because a null that reaches the scorer becomes a zero somewhere, and a zero sorts.
Convert units at the boundary, never downstream
Funding intervals are annualised inside the adapter that knows the interval. Timestamps become UTC instants there too. If a unit conversion happens anywhere else, it will eventually happen twice or not at all.
Carry provenance on every record
Source, upstream identifier, fetch time, and adapter version ride along. When a number looks wrong at 3am, provenance is the difference between reproducing it in a minute and guessing for an hour.
Run one shared conformance suite across every adapter
The same fixture battery — missing field, wrong type, unexpected null, unknown symbol, clock skew — runs against all of them. A new adapter isn’t finished when it works; it’s finished when it fails the same way the others do.
Where this runs
CryptoFundingWatch
Five exchange adapters behind one instrument model. The scoring engine ranks by annualised spread and has no knowledge that Kraken calls Bitcoin XBT or that Hyperliquid quotes hourly. Adding a sixth venue is an adapter and a fixture set.
Tests: uniform cadence, numeric precision, symbol mapping
LatticePulse
Four adapters producing a typed attention event rather than a single number — ticker, instant, kind, provenance, and the source-native payload. The scorer applies per-kind weights, so a Form 4 purchase and a Reddit thread stay distinguishable all the way through.
Tests: mixed media types, 15–30 min vs minute-level polling
What the bench showed
Normalise identity and time; do not normalise meaning. This is the difference between the two systems, and it took building both to see it. CryptoFundingWatch should collapse five rates into one comparable number, because they genuinely measure the same quantity. LatticePulse should not, because a Reddit score and an insider purchase do not. The canonical model’s job is to agree on which ticker and when — agreeing on how much it matters is the scorer’s job, and moving it earlier destroys information.
Validation at the boundary converts a silent corruption into a visible outage. When the drift button renames a field, the affected source drops out and flags itself while the rest keep producing. Without that check the adapter emits nulls, the scorer reads them as zeros, and the venue quietly ranks bottom — a wrong answer that looks like a right one.
Adapter purity is what makes the conformance suite possible, and the conformance suite is what makes adding a source boring. That’s the actual goal: the sixth exchange should be a dull afternoon, not a release.
Bench source: public/schemas/index.html ·
Related: bench 03 — retry & dead letter