scaling an oracle from hundreds of data points to 100 billion
how to get a supply chain onto a blockchain without putting it all there
commits a huge off-chain dataset to a chain as one cryptographic root. hundreds of data points to an estimated 100 billion.
[code]
commodity traceability on-chain has a scale problem that kills naive designs. a shipment of cocoa is thousands of events. harvest, weigh, move, certify. a real supply chain is millions of those. a conventional oracle puts each data point on-chain, so it tops out at tens, maybe hundreds, because every point costs space and gas. you cannot put a supply chain on a blockchain one row at a time.
the job was to get the on-chain footprint of a huge dataset down to something a chain can actually verify, without giving up integrity. this is the piece i point to as my most interesting build.
the move is to stop storing the data on-chain and store a commitment to it instead. the chain holds one cryptographic root, a merkle-trie root. the bulk data and the proofs live off-chain. to prove any single data point, you hand over a proof that verifies against the on-chain root. tamper with one leaf and the root changes, so the commitment is worth exactly as much as putting everything on-chain, at a fixed size.
the actual data needs to be saved somewhere like a database, but no one has to take your word on the integrity. they get the data from you, then can recompute the path back up the trie and check that it lands on the same root the chain already knows about. if it matches, the data is part of the committed dataset. if it does not, either the data or the proof is fake.
the database is allowed to be boring, cheap, centralized, replicated, compressed, indexed, whatever. it is only responsible for availability. the chain is responsible for integrity. on-chain cost stops growing with the dataset.
the leaves are illustrative, two shipments of harvest, weigh, move, certify. only the 32 byte root goes on-chain. select a leaf to light its proof path.
the leaves are illustrative, two shipments of harvest, weigh, move, certify. only the 32 byte root goes on-chain. select a leaf to light its proof path.
i built the oracle node in Go on an actormodel architecture with postgres for state and IPFS for blob storage, behind a REST API that returns proofs for individual points. a few things mattered more than they sound. canonicalization has to be deterministic, so numbers get turned into rational strings and two nodes always hash the same data to the same root. ingest has to be rollback safe, because chains reorg (two miners mining the same block number creating a chain split which needs to be reconciled) and a rollback should rewind trie state deterministically. and the cardano providers are pluggable behind one interface, because no single data source is reliable enough to bet a protocol on (especially on cardano).
the system went from handling tens or hundreds of data points to an estimated 100 billion, because the on-chain cost is a fixed-size root instead of something that grows with the data.
the node is public: merkle-oracle-node.
https://github.com/zenGate-Global/merkle-oracle-node
what i actually took away is that traceability is mostly instrumenting messy physical labor into structured, verifiable data. and both are just as challenging.