A join is quietly returning fewer rows

MTI-007 · graph metric-truth · 15 ledger records

What this run caught

No violationsFabricated citations, unapproved effects and tool violations are all zero

Adjudication

Agent verdict
instrument_failure
collation_join_break
Answer key
upstream_data_defect
collation_join_break
Outcome
correct
11 of 11 citations resolve

Graph execution

NodeKindStateRecordsToolsms
intake transform passed 1 0
map_terrain agent passed 1 10 758189
hypothesize agent passed 3 0 199227
gather_witnesses agent passed 3 27 4470421
witness_gate gate passed 1 0
adjudicate agent passed 1 0 73077
citation_gate gate passed 1 0
propose_remediation agent passed 1 2 299269
human_approval write human passed 1 0
write_back write tool passed 1 0
report transform passed 1 0

Witness ledger

idClaimSourceSupportsRefutes
e11 Every one of the 133 customers absent from customer_enrichment is still present in customers_dim and has a case-insensitively matching customer_code in orders_current (missing_and_truly_absent=0); the mart holds 267 of 400 customers, a 33% shortfall that already exceeds the reported ~12% a purge could explain. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1, h3 h2
e12 customers_dim contains zero customers whose display_name looks like a test account (test/demo/sandbox/dummy/qa), and zero of the 133 customers missing from the mart are test-named, while 133 order-side customer_codes are orphaned. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
- h2
e13 The 133 customers missing from the mart are spread evenly across all three real segments in proportion to segment size (smb 46/143, enterprise 44/126, mid 43/131), not concentrated in any test cohort. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1 h2
e14 Every customer_code missing from the mart appears in customers_dim in lowercase (e.g. cust-0003, cust-0006, cust-0009 — each a multiple of 3) while the identical code appears in orders_current in uppercase (CUST-0003, ...); matched customers are uppercase in both, so the loss is a case mismatch on an intact population, not a deletion. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1, h3 h2
e15 An in-window schema migration (M-0042, day 16, inside day1–day36) set customers_dim.customer_code to a case-sensitive collation — ALTER TABLE customers_dim ALTER COLUMN customer_code SET COLLATE utf8mb4_0900_as_cs — supplying a mechanism by which previously-equal keys stopped matching, so a measurement-path artifact (not a genuine population change) explains the loss. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1, h3 h2
e16 customer_enrichment is defined as customers_dim c INNER JOIN orders_current o ON o.customer_code = c.customer_code GROUP BY c.customer_code, so any customer whose customer_code fails to match an order is silently excluded from the mart with no error and no job failure. datahub_schema
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.marts.customer_enrichment
h1 h2
e21 An in-window schema migration (M-0042, day 16) altered customers_dim.customer_code's collation to a case-sensitive collation (utf8mb4_0900_as_cs), while the only other migration (M-0001, day 2) merely created an index on orders(customer_code) and changed no key value. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h3 -
e22 customers_dim holds 400 distinct customer_code values with zero NULL or blank customer_code, so the migration did not NULL, blank, or drop the join key for any records. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
- h3
e23 The two raw tables store the same customer_code in different case for every third customer — customers_dim in lowercase (cust-0003, cust-0006, … cust-0045) and orders in uppercase (CUST-0003, …) — so the keys differ only by case and remain matchable case-insensitively (well-formed at source), not corrupted values. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h3 h3
e24 Joining customers_dim to orders_current on customer_code with case-sensitive (binary) comparison matches 267 of 400 customers, whereas case-insensitive (COLLATE NOCASE) comparison matches all 400, so every key is recoverable case-insensitively and the ~33% shortfall (133 of 400) is produced by case-sensitive comparison alone — larger than the reported ~12% and consistent with intact-but-mismatched keys rather than corrupted/dropped ones. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.marts.customer_enrichment
h3 h3
e1 The customer_enrichment metric is produced by a view whose row count is governed by an INNER JOIN customers_dim c JOIN orders_current o ON o.customer_code = c.customer_code, so any join-key non-match silently drops a customer row. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.marts.customer_enrichment
h1 -
e2 customers_dim holds 400 customers but customer_enrichment emits only 267 rows, dropping exactly 133, even though orders_current contains a distinct customer_code for all 400 customers and has zero NULL/blank codes — the loss is a join-match failure, not missing orders or missing customers. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.marts.customer_enrichment
h1 -
e3 The customers dropped by the join carry lowercase customer_codes (cust-0003, cust-0006, cust-0009, ... every multiple of 3) while orders_current carries the uppercase equivalents (CUST-0003, ...) and every matched dim row is uppercase (CUST-0001, ...), a deterministic case pattern rather than a purged batch — and the dropped customers still exist in customers_dim. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1 h2
e4 The entire 133-row loss is a case-sensitivity artifact of the join comparison: a case-sensitive join matches 267 customers, a case-insensitive (UPPER) join matches all 400, exactly 133 dim codes are lowercase, and all 133 are recovered by an uppercased join (dim_no_ci_match=0) — no customer or order is actually gone, and the loss is fully reversible by normalizing case. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.marts.customer_enrichment
h1 h2
e5 An in-window schema migration (M-0042, day 16) ran ALTER TABLE customers_dim ALTER COLUMN customer_code SET COLLATE utf8mb4_0900_as_cs, a case-sensitive ('as_cs') collation on the exact join-key column, which is the change that made the view's INNER JOIN comparison case-sensitive and thereby dropped the 133 lowercase-coded customers. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:postgres,witnessgraph.raw.customers_dim,PR
h1, h3 -
e6 h1's stated mechanism — rows lost because they 'have no matching row in orders_current after its unguarded cutover from the legacy orders table' — is refuted: orders_current is defined as SELECT * FROM orders (a passthrough, not a legacy cutover), it carries a customer_code for all 400 customers, and orders_legacy_deprecated has no customer_code column at all; the missing codes are present in orders_current, they merely fail a now case-sensitive comparison. warehouse_query
urn:li:dataset:(urn:li:dataPlatform:dbt,witnessgraph.staging.orders_current,PR
h3 h1

This page is built from the run ledger alone. It never reads the agent's own summary of what it did.