8000 feat(dq): deterministic check engine + cross-dialect portability + churn fix by BaraaZ95 · Pull Request #21 · zaherab/DQOPS_API · GitHub
[go: up one dir, main page]

Skip to content

feat(dq): deterministic check engine + cross-dialect portability + churn fix - #21

Merged
zaherab merged 5 commits into
mainfrom
fix/unrunnable-check-hardening
May 22, 2026
Merged

zaherab merged 5 commits into
mainfrom
fix/unrunnable-check-hardening

Conversation

@BaraaZ95
Copy link
Copy Markdown

Summary

Replaces the LLM-based DQ check recommender with a fully deterministic
engine, makes sensor SQL portable across 5 dialects, and fixes the
check-churn / stale-score defects found in end-to-end trigger testing.

Five commits over main:

  • feat(dq): deterministic check engine + cross-dialect sensor portability — profile aggregates + bounded sampling + pure-Python inference/threshold/emit. sqlglot transpilation; per-dialect sensor templates.
  • chore(docker): pin python 3.12 — 3.14 was unintended.
  • fix(security): escape regex_pattern in sensor SQL — blocks SQL injection via regex check params.
  • fix(dq): make check_type → dimension a single source of truth — EmittedCheck.dimension is derived from check_type; /checks/types exposes the resolved dimension. Removes the emitter↔orchestrator disagreement that caused infinite check create/delete churn.

Testing

  • Full deterministic-engine suite (tests/profilers, tests/connectors, check routes, dimension mapping) — 144/144 on touched files.
  • New tests: structural-check gating, an 8-dimension "no check emitted outside the promised set" invariant, conformity routing for range/length checks, /checks/types dimension exposure.
  • Verified end-to-end against MLG-Spec-Reader: 9/9 data products idempotent (no churn), all DQ triggers green.

🤖 Generated with Claude Code

BaraaZ95 and others added 5 commits May 21, 2026 16:14
Add a deterministic DQ check engine and make check execution portable
across PostgreSQL, DuckDB, MySQL, SQL Server, and Oracle.

Engine (profilers/):
- profile_runner: single-round-trip aggregate profiling, opaque aliases.
- sample_fetcher: bounded hash-mod row sampling, PII columns excluded.
- sample_cache: Redis-backed, shared across workers, fingerprint-invalidated.
- inference_engine: regex/codelist/enum/format/range detection with
  bundled ISO 3166 + ISO 4217 codelists.
- threshold_engine: promise -> rule_parameters projection (Python port
  of MLG dq-promise-thresholds.ts, parity-locked via shared fixtures).
- check_emitter: deterministic rules -> calibrated check specs, with
  not_assessed reason codes for promised-but-uncoverable dims.

Dialect portability:
- Connectors gain build_profile_sql / build_sample_sql /
  get_schema_fingerprint with per-dialect hash, cast, length, limit.
- Sensor SQL transpiled with sqlglot at execution; sensors declare
  dialect_templates and unsupported_dialects (capability matrix ->
  clean SensorUnsupportedError skip, recorded not_assessed).
- In-set sensors rewritten from PG ANY(ARRAY[...]) to portable IN(...);
  country/currency code validate against real ISO codelists.
- ODPS dimension mapping re-aligned to spec: numeric/range -> conformity,
  referential -> accuracy, anomaly excluded.

API + ops:
- POST /v1/profile/run and POST /v1/checks/emit-deterministic.
- /metrics endpoint + counter registry.
- profile_audit logs every customer-DB query (hash, size, duration).

Security: in-set expected_values escaped before templating to prevent
SQL injection via producer-controlled acceptedValues.

Dockerfile: add msodbcsql18 and a shared-data mountpoint.

Tests: 313 passing; 9 dialects syntax-verified, 5 verified live.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3.14 is too new — C-extension wheels for duckdb, oracledb, pyodbc, and
great-expectations are not all published yet, so the image build is
fragile. 3.12 is the supported floor (requires-python = ">=3.12") and
has full wheel coverage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
regex_pattern renders inside a SQL string literal ('{{ regex_pattern }}')
in the four regex sensors. It is producer-controlled — sourced from the
contentSchema `pattern` declaration or an inferred regex — and Jinja does
not escape SQL quotes. A pattern containing a single quote breaks out of
the literal (SQL injection), same class as the in-set expected_values
bug fixed earlier.

Sensor.render now doubles single quotes in string-literal params
(regex_pattern, expected_type) before templating. Regression test added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The deterministic emitter and the MLG orchestrator each resolved a
check's ODPS dimension independently, and disagreed:

- The emitter hardcoded a dimension per branch. number_in_range_percent
  and text_length_in_range_percent were labelled "validity", but
  get_dimension_for_check_type() — used for scoring and orphan detection
  — maps both to "conformity" (range/length are conformity standards).
- column_count / column_list_changed were emitted unconditionally,
  regardless of whether their dimension was promised.

Either disagreement made the orchestrator treat the check as an orphan
(dimension not promised) and delete it, then re-emit it next run — an
infinite create/delete churn.

Fixes:
- EmittedCheck.dimension is now derived from check_type in __post_init__
  via get_dimension_for_check_type(). Emit sites no longer pass it; a
  wrong dimension is structurally impossible.
- Every emit gate, threshold-promise lookup, and structural-check
  emission flows from _dim_for() — the same canonical map.
- /checks/types now returns the resolved `dimension` per check type, so
  external orchestrators consume it directly instead of re-deriving from
  `category` (which has no entry for `schema` and misses overrides).

Adds tests: structural-check gating, an 8-dimension "no check emitted
outside the promised set" invariant, conformity routing for range/length
checks, and /checks/types dimension exposure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zaherab
zaherab merged commit d581956 into main May 22, 2026
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0