Repository navigation
feat(dq): deterministic check engine + cross-dialect portability + churn fix - #21
Merged
Merged
Conversation
Add a deterministic DQ check engine and make check execution portable across PostgreSQL, DuckDB, MySQL, SQL Server, and Oracle. Engine (profilers/): - profile_runner: single-round-trip aggregate profiling, opaque aliases. - sample_fetcher: bounded hash-mod row sampling, PII columns excluded. - sample_cache: Redis-backed, shared across workers, fingerprint-invalidated. - inference_engine: regex/codelist/enum/format/range detection with bundled ISO 3166 + ISO 4217 codelists. - threshold_engine: promise -> rule_parameters projection (Python port of MLG dq-promise-thresholds.ts, parity-locked via shared fixtures). - check_emitter: deterministic rules -> calibrated check specs, with not_assessed reason codes for promised-but-uncoverable dims. Dialect portability: - Connectors gain build_profile_sql / build_sample_sql / get_schema_fingerprint with per-dialect hash, cast, length, limit. - Sensor SQL transpiled with sqlglot at execution; sensors declare dialect_templates and unsupported_dialects (capability matrix -> clean SensorUnsupportedError skip, recorded not_assessed). - In-set sensors rewritten from PG ANY(ARRAY[...]) to portable IN(...); country/currency code validate against real ISO codelists. - ODPS dimension mapping re-aligned to spec: numeric/range -> conformity, referential -> accuracy, anomaly excluded. API + ops: - POST /v1/profile/run and POST /v1/checks/emit-deterministic. - /metrics endpoint + counter registry. - profile_audit logs every customer-DB query (hash, size, duration). Security: in-set expected_values escaped before templating to prevent SQL injection via producer-controlled acceptedValues. Dockerfile: add msodbcsql18 and a shared-data mountpoint. Tests: 313 passing; 9 dialects syntax-verified, 5 verified live. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hardening # Conflicts: # Dockerfile
3.14 is too new — C-extension wheels for duckdb, oracledb, pyodbc, and great-expectations are not all published yet, so the image build is fragile. 3.12 is the supported floor (requires-python = ">=3.12") and has full wheel coverage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
regex_pattern renders inside a SQL string literal ('{{ regex_pattern }}')
in the four regex sensors. It is producer-controlled — sourced from the
contentSchema `pattern` declaration or an inferred regex — and Jinja does
not escape SQL quotes. A pattern containing a single quote breaks out of
the literal (SQL injection), same class as the in-set expected_values
bug fixed earlier.
Sensor.render now doubles single quotes in string-literal params
(regex_pattern, expected_type) before templating. Regression test added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The deterministic emitter and the MLG orchestrator each resolved a check's ODPS dimension independently, and disagreed: - The emitter hardcoded a dimension per branch. number_in_range_percent and text_length_in_range_percent were labelled "validity", but get_dimension_for_check_type() — used for scoring and orphan detection — maps both to "conformity" (range/length are conformity standards). - column_count / column_list_changed were emitted unconditionally, regardless of whether their dimension was promised. Either disagreement made the orchestrator treat the check as an orphan (dimension not promised) and delete it, then re-emit it next run — an infinite create/delete churn. Fixes: - EmittedCheck.dimension is now derived from check_type in __post_init__ via get_dimension_for_check_type(). Emit sites no longer pass it; a wrong dimension is structurally impossible. - Every emit gate, threshold-promise lookup, and structural-check emission flows from _dim_for() — the same canonical map. - /checks/types now returns the resolved `dimension` per check type, so external orchestrators consume it directly instead of re-deriving from `category` (which has no entry for `schema` and misses overrides). Adds tests: structural-check gating, an 8-dimension "no check emitted outside the promised set" invariant, conformity routing for range/length checks, and /checks/types dimension exposure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the LLM-based DQ check recommender with a fully deterministic
engine, makes sensor SQL portable across 5 dialects, and fixes the
check-churn / stale-score defects found in end-to-end trigger testing.
Five commits over
main:sqlglottranspilation; per-dialect sensor templates.EmittedCheck.dimensionis derived fromcheck_type;/checks/typesexposes the resolveddimension. Removes the emitter↔orchestrator disagreement that caused infinite check create/delete churn.Testing
tests/profilers,tests/connectors, check routes, dimension mapping) — 144/144 on touched files./checks/typesdimension exposure.🤖 Generated with Claude Code