Data Membrane enforces contracts on data crossing your boundary, in either direction. Coming in, partner and vendor feeds are checked against your contract before they reach your systems. Going out, regulatory and partner submissions are checked against their published spec before they're filed. Every record is accounted for, and every failure is tied to the rule it broke.
Every removed row is attributed to the stage and the rule that removed it, and the arithmetic has to foot. Delivery refuses to write when it doesn't. The file is generated test data with defects seeded at known counts, and every count matched. One run, not an averaged benchmark — we'll show you the log.
On a Federal Reserve return (FR Y‑14M Schedule D.1: 118 columns, 188 rules), every seeded defect was caught by the rule that should catch it, none by the wrong rule and none invented, and repeated 5M‑row runs reconciled exactly.
Most quality tooling runs after the load. It tells you on Tuesday that Monday's file carried four hundred thousand rows with a null settlement ID — by which point those rows are in your lake, in your marts, and inside a figure someone has already acted on.
Going out, it's worse. An error in a regulatory return is often found by the regulator — after it's filed, with someone accountable's name on it.
Removing bad rows is a project. Refiling is a conversation with your regulator. And the next file arrives tomorrow.
The answer isn't better detection after the fact. It's a boundary: a place where data has to prove it conforms before it is allowed in, or allowed out, and a record of exactly what was held and why.
Vendor files, partner extracts, subsidiary feeds. Checked against your contract before they reach your lake or warehouse; what fails is held with evidence you can send back to whoever produced it.
Regulatory and partner submissions, checked against the published spec before they're filed. We validate the dataset the return is built from; your existing tagging and filing step stays where it is.
You author one contract. The division below is ours to manage, not yours to configure.
Schema conformance, casting and rejection happen at ingestion. A record whose types can't be realised against your declared schema never enters the pipeline at all. Nothing downstream has to defend against it.
Row rules are evaluated in Polars, batch rules in DuckDB. A record that is structurally valid but semantically wrong — a negative amount, a currency you don't settle in, a date that precedes its own predecessor — is held here.
Data Membrane has no opinion about whether twelve million held rows is a disaster or an ordinary Tuesday for that feed.
If you want a run to stop when too little survives, you declare it — MIN_ROW_COUNT is the absolute form. If you want it to proceed and report, it proceeds and reports.
A terminal status of LOADED is a statement of fact about what was delivered, not a verdict on your data. FAILED is reserved for our faults.
Row rules compiled once per run and evaluated columnar, with one meaning for every expression on every engine.
Batch rules and stop conditions over the whole run, with a bounded spill so a large run fails cleanly rather than filling a disk.
One columnar representation from ingestion to delivery, passed between stages by reference rather than through the message bus.
Runs on k3s or your own cluster. Every image runs as non-root, and the data plane's pods meet the restricted security profile.
The control plane holds contracts, versions and run history. The data plane reads and enforces your data. You choose where the data plane lives.
We create and maintain the cluster in your account, under a least-privilege role you can audit and revoke. Client credentials live only in the data plane; the control plane never holds them.
The data plane runs on our infrastructure, for teams that would rather not run anything. The same boundary, the same evidence, the same contract.
On either topology the data plane dials out and nothing dials into it; the message bus is authenticated with per-service permissions; and the control plane receives counts, rule outcomes and contract versions, not your records in bulk. One thing we'd rather volunteer: the managed-in-your-cloud option has not yet run at a client. The first design partner who chooses it is the first.
Version one is built, hardened and in testing with design partners. We'd rather tell you all of this now than have you find it during a pilot.
Object storage to Delta Lake is the path tested hardest: 30M rows delivered to remote storage in one run, reconciliation matched. PostgreSQL reads a named table and writes a run in one transaction.
A contract naming one of these is refused when it's saved, rather than failing at run time. Warehouse targets open for the first design partner who needs one. Streaming is refused by design: a run reconciles against an end, and a stream has none.
Every contract is versioned as it changes. You can retrieve any prior version, or the version that was live at a given moment.
Structural rejection at ingestion, row and batch rules at quality, and a completeness check that stops delivery when the row arithmetic doesn't reconcile.
Each quality rule carries a stable id, kept across contract versions. Violations, rejection reports and held records all name the rule by that id.
The message bus runs over TLS with per-service authentication and permissions, the chunk store checks a signed token on every request, the control plane is reached over TLS, and client credentials never leave the data plane.
Four roles and nine capabilities, enforced across every control-plane route. Your operator manages users and passwords without us; single sign-on isn't built yet.
Each client gets a dedicated deployment — a cost we carry, not you. Shared tenancy waits until a client asks for shared pricing.
Contracts can name upstream references; nothing resolves them into a graph yet. If lineage-driven orchestration is central to your case, we want to scope it with you rather than claim it.
A rule today judges one run. Checks across schedules of the same return, or against last period's figures, are designed but not built — and we know regulators care about them.
Our format borrows from the Open Data Contract Standard but is our own. Until the exporter lands, your contracts are portable as JSON, not as a standard.
The two items we listed in August, with how they came out, and the next thing we're building.
Repeating it found a real limit: the old write held memory for the whole run and failed near 29.5M rows. Delivery now writes the files itself and commits them once — 30M rows to remote storage in 3 min 25 s, reconciliation matched.
Running on two hosts since mid-September. The data plane dials out and nothing dials in, asserted by test in both directions; the bus between them is now authenticated.
Supply a published specification and get back a contract whose every rule cites the passage it came from. Designed and decided, not yet built — the runtime above is what it will feed.
The best fit is data you can't fix at the source — a vendor file, a partner extract coming in, or a regulatory return going out — where the cost of getting it wrong is real and someone can name it.
Three fields. We reply within two working days, or we tell you we're full.