8000 feat(lint): add UI selector contract lint for nightly verifiers by VLukiantsev · Pull Request #1626 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content

feat(lint): add UI selector contract lint for nightly verifiers - #1626

Open
VLukiantsev wants to merge 29 commits into
feat/ui-selector-data-testidfrom
feat/ui-selector-contract-lint
Open

VLukiantsev wants to merge 29 commits into
feat/ui-selector-data-testidfrom
feat/ui-selector-contract-lint

Conversation

@VLukiantsev
@VLukiantsev VLukiantsev commented Nov 9, 2026 •
Copy link
Copy Markdown
Collaborator

Summary

Adds tools/lint/check_ui_selector_contract.py, a pre-commit and CI lint that enforces the data-testid selector contract between the Python nightly UI verifiers (tools/nightly_tests/) and the Angular templates (cmd/ui/app/**). A template change that removes a data-testid or component tag a verifier depends on, or a verifie 8000 r that couples to non-framework CSS classes, non-standard attributes, or user-facing copy without an explicit # ui-contract: <reason> annotation, now fails at commit time instead of in the 04:00 UTC nightly run.

Motivation & Context

#1522 changed the ROD queue jobs table to render the experiment title, and the nightly ROD verifier matched rows with has_text=f"Experiment #{xid}". Every presubmit passed because the verifier has no build-graph edge to cmd/ui/app/**, its unit test mocks the DOM, and the KinD E2E never visits /resources.

Following the design review on #1626:

  • Standardizing nightly verifier selectors on explicit data-testid attributes (feat(ui): migrate nightly verifier selectors to data-testid #1643) avoids regex-indexing class=, [class.x], and [ngClass] across Angular templates and decouples E2E verification from CSS styling refactors.
  • Non-Material CSS classes, non-standard attributes, and :has-text / has_text= copy matches are disallowed unless accompanied by an explicit # ui-contract: <reason> annotation on the selector line or a comment-only line immediately above.

Key Changes

  • New tools/lint/check_ui_selector_contract.py (+ test, Bazel targets).
    • Tracked-file discovery: enumerates tracked files via git ls-files -z -- tools/nightly_tests and git ls-files -z -- cmd/ui/app cmd/ui/index.html (matching check_file_naming.py), so untracked scratch files and local venvs are ignored. Any syntax error in a tracked nightly file fails the lint, and EXCLUDED_SOURCES lets files opt out (empty today). Collects *_SEL / *_SEL_TMPL constants (including tuple/list unpacking and annotated assignments) and the first positional or selector= argument of .locator() / .wait_for_selector(). {...} placeholders are wildcards.
    • dom-testid: every [data-testid="..."] value must match a literal data-testid="..." attribute in cmd/ui/app/**/*.ng.html or cmd/ui/index.html (bare [data-testid] without a literal value also fails).
    • dom-tag: a custom element (with -) must be a declared Angular component selector: in cmd/ui/app/**/*.ts (skipping *_test.ts and *_harness.ts) or appear as a tag in a template.
    • dom-class / dom-attribute: non-Material classes and non-standard attributes fail unless the selector carries # ui-contract: <reason>.
    • Exemptions: Material/CDK names (mat-, mdc-, cdk- tags, classes, and attributes) and the href, disabled, aria-* attributes.
    • copy: :has-text("…") and has_text="…" literals need # ui-contract: <reason> on the same line or a comment-only line immediately above.
    • unknown-selector-constant: every alias.NAME or imported name taken from ui_css_selectors must be a *_SEL / *_SEL_TMPL constant collected from that module, enforcing the naming convention.
    • Output: JSON; exit codes 0/1/2/130.
  • Wiring:
    • .githooks/pre-commit runs the lint on every commit after check_file_naming (~0.2 s; no path gate, avoiding duplicate scope lists that could drift silently).
    • lint.yml gets an unconditional "Check Nightly UI Selector Contract" step.
  • Docs: tools/README.md index entry; one line in tools/nightly_tests/README.md.

Assumptions

  • Material/CDK names (mat-, mdc-, cdk-) are exempt by prefix — they are generated by Angular Material at runtime and never appear as static data-testid attributes in templates — if wrong, a Material DOM change after a dependency bump is still caught only by nightly.
  • Only :has-text(...) and has_text= copy matches are linted — they are the selector-level pattern behind feat(rod): display experiment title in queue jobs table #1522 — if wrong, has_not_text= (one use today, an em-dash placeholder check) and post-hoc text_content() assertions are still caught only by nightly.
  • Out of scope: the KinD /resources E2E scenario, resolve_test_scope.py mapping, and proto/enum parity checks — separate slices of the same tracking bug — if wrong, coverage of those couplings stays nightly-only until they land.

Verification & Testing

  • Commands Executed:
    • bazel test //tools/lint/...
    • ./.githooks/pre-commit
    • python3 tools/lint/check_ui_selector_contract.py
  • Coverage & Test Status:
    • Bazel: 8/8 targets pass.
    • Lint on the current tree: success (6 sources, 77 selectors, 0 violations).
    • Unit tests cover:
      • tuple/list-unpacked and annotated constants, selector= keywords, implicit concatenation, f-strings, and placeholders
      • dom-testid matches, missing testids, and bare [data-testid] without a literal value
      • non-Material classes and non-standard attributes failing unless annotated with # ui-contract:
      • *_test.ts and *_harness.ts exclusion from component selector indexing
      • comment-only line-above annotation rule (trailing # ui-contract: on a preceding code line does not bleed into the next selector)
      • git ls-files -z tracked-file enumeration (untracked broken files ignored; non-git --repo-root handled)
      • discovery of a verifier at an arbitrary path, has_text=-only sources, EXCLUDED_SOURCES, and unknown-selector-constant via module alias, relative import, import … as, and from … import (X, Y as Z)

Related Issues / Bugs

Part of bug 581241116 (slice: UI selector contract lint). Stacked on #1643; follows #1624 (merged).

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024

…ct lint

The verify step in nightly-examples-e2e.yml stopped forwarding
--expected_work_units after nightly-test-suite.yml was deleted (#1206),
so verifiers silently fell back to hardcoded defaults. The ROD UI
verifier and the simple_job duration tag were hardcoded outright.

- Matrix entries launching examples/simple_job declare
  expected_work_units and expected_duration_sec; the verify step
  forwards them via env.
- verifier_cli gains --expected_duration_sec; simple_job and ROD
  verifiers derive expectations from the resolvers.
- New tools/nightly_tests/check_matrix_contract.py enforces that launch
  args equal the expected_* fields and that the verify step forwards
  them; wired into .githooks/pre-commit and lint.yml.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on check_matrix_contract.py:
- A space-form contract flag followed by another `--` flag resolves
  to an empty value instead of consuming the next flag.
- Blank and comment lines no longer end the verify step block.
- The verify step start must be indented less than its `id:` line,
  so nested list items in run blocks are skipped.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
- `_VERIFY_STEP_ID_RE` accepts `- id: verify-example` as the first step
  key.
- When the id line itself opens the step, it is the step start; no
  backward search that could match an outer list item.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Nightly UI verifiers reference Angular DOM through hand-copied
selector strings with no build-graph edge to cmd/ui/app, so template
changes (e.g. #1522) pass presubmit and break nightly.

check_ui_selector_contract.py collects selectors from the nightly
verifier sources via ast and requires every custom tag, class and
attribute to exist in the Angular templates/components (Material/CDK
prefixes exempt). Literal :has-text / has_text copy matches require a
`# ui-contract: <reason>` annotation; existing matches are annotated.
Wired into .githooks/pre-commit and lint.yml.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new static analysis tool, check_ui_selector_contract.py, along with its corresponding unit tests and Bazel configuration. The tool validates that nightly UI verifier selectors (Playwright CSS selectors) match tags, classes, and attributes declared in Angular templates, and enforces that any copy-matching selectors are annotated with a # ui-contract comment. The pre-commit hooks and documentation have been updated to integrate this check, and existing test files have been annotated accordingly. The review feedback focuses on improving the robustness of the linter by supporting single quotes in templates, handling list unpacking, supporting keyword arguments in selector calls, and guarding directory lookups with safety checks.

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new static analysis linter, check_ui_selector_contract.py, along with associated unit tests, Bazel build targets, and pre-commit hooks. The linter ensures that nightly UI verifier CSS selectors remain synchronized with Angular templates, requiring explicit # ui-contract: annotations for text-based matches. The feedback suggests enhancing the linter's robustness by updating the regular expressions for static classes and class bindings to support both single and double quotes, preventing potential false positives.

Address review feedback on check_ui_selector_contract.py:
- Static `class` and `[class]`/`[ngClass]` bindings accept single or
  double quotes via a matching-quote backreference.
- Selector constants may use list as well as tuple unpacking.
- `selector=` keyword arguments to locator/wait_for_selector are
  collected.
- A missing cmd/ui/app directory is a handled error instead of
  reporting every selector as missing.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new linter tool, check_ui_selector_contract.py, along with its unit tests, to ensure that nightly UI verifier selectors remain synchronized with Angular templates. It checks for undeclared DOM tags, classes, and attributes, and enforces annotations for text-matching selectors. The pre-commit hooks, Bazel build files, and documentation have been updated accordingly, and existing UI tests have been annotated to comply with the new contract. The review feedback suggests improving the TypeScript selector regex to support backticks and removing the strict keyword from zip() to ensure compatibility with Python versions older than 3.10.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
VLukiantsev added this pull request to stack #1629 November 9, 2026 15:58
The pre-commit regex and lint.yml path filter duplicated
SELECTOR_SOURCES in two other syntaxes with no parity check, so a new
or renamed selector source would silently skip the gate. The lint
takes ~0.2s; run it on every commit and CI lint run instead, matching
check_matrix_contract and check_file_naming.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Replace the hand-kept SELECTOR_SOURCES glob list, which failed open
for new or renamed verifiers, with ast discovery over every *.py under
tools/nightly_tests (hidden dirs and __pycache__ skipped; syntax errors
fail the lint; EXCLUDED_SOURCES opt-out). Add an
unknown-selector-constant rule: every reference to a ui_css_selectors
name must be a collected *_SEL/*_SEL_TMPL constant, so the naming
convention is enforced rather than assumed.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Base automatically changed from fix/nightly-expected-work-units to main November 10, 2026 12:10
Nightly UI verifiers reference Angular DOM through hand-copied
selector strings with no build-graph edge to cmd/ui/app, so template
changes (e.g. #1522) pass presubmit and break nightly.

check_ui_selector_contract.py collects selectors from the nightly
verifier sources via ast and requires every custom tag, class and
attribute to exist in the Angular templates/components (Material/CDK
prefixes exempt). Literal :has-text / has_text copy matches require a
`# ui-contract: <reason>` annotation; existing matches are annotated.
Wired into .githooks/pre-commit and lint.yml.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on check_ui_selector_contract.py:
- Static `class` and `[class]`/`[ngClass]` bindings accept single or
  double quotes via a matching-quote backreference.
- Selector constants may use list as well as tuple unpacking.
- `selector=` keyword arguments to locator/wait_for_selector are
  collected.
- A missing cmd/ui/app directory is a handled error instead of
  reporting every selector as missing.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
The pre-commit regex and lint.yml path filter duplicated
SELECTOR_SOURCES in two other syntaxes with no parity check, so a new
or renamed selector source would silently skip the gate. The lint
takes ~0.2s; run it on every commit and CI lint run instead, matching
check_matrix_contract and check_file_naming.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Replace the hand-kept SELECTOR_SOURCES glob list, which failed open
for new or renamed verifiers, with ast discovery over every *.py under
tools/nightly_tests (hidden dirs and __pycache__ skipped; syntax errors
fail the lint; EXCLUDED_SOURCES opt-out). Add an
unknown-selector-constant rule: every reference to a ui_css_selectors
name must be a collected *_SEL/*_SEL_TMPL constant, so the naming
convention is enforced rather than assumed.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
VLukiantsev force-pushed the feat/ui-selector-contract-lint branch from c9ec2e2 to 2692b16 Compare November 10, 2026 12:10
@VLukiantsev
VLukiantsev marked this pull request as ready for review November 10, 2026 12:20
@github-actions
github-actions Bot requested a review from janzak-dev November 10, 2026 12:21
Comment thread .githooks/pre-commit Resolved
@janzak-dev
Copy link
Copy Markdown
Collaborator

Design Question: data-testid contract vs. regex-indexing all Angular templates

Before investing further in maintaining template regexes: check_ui_selector_contract.py (~356 lines, plus ~311 lines of tests) scans every *.ng.html and non-test *.ts file under cmd/ui/app (plus cmd/ui/index.html) with regexes to build global, unscoped sets of tags, classes, and attributes, and runs unconditionally on every commit.

Because the index is global and regex-based rather than a real Angular/CSS parser, it only proves a token name exists somewhere in the UI (not on the target component), while adding substantial regex surface area to keep in sync with Angular template syntax.

Would it be simpler and less fragile to standardize nightly verifier selectors on explicit data-testid attributes (exempting Material/CDK selectors or requiring # ui-contract: for exceptions)? That turns the check into a small literal match between data-testid values in tools/nightly_tests and cmd/ui/app templates, and avoids coupling nightly tests to styling classes or template formatting.

Add explicit `data-testid` attributes to the 20 Angular component
templates exercised by the Python nightly UI verifiers (plus
`[attr.aria-pressed]` on the velocity period toggle pills) and update
`ui_css_selectors.py` and `ui_entities_validator.py` to query
`[data-testid="..."]` instead of styling classes.

This decouples nightly E2E selectors from CSS class names and template
formatting ahead of the `check_ui_selector_contract.py` lint gate in
#1626.

Part of bug 581241116

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on #1626 ( stacked on #1643 ):
- Replace CSS class and general attribute template regex indexing with
  a literal `data-testid="..."` index (`dom-testid`); non-Material
  classes and non-standard attributes in nightly selectors now require
  `# ui-contract: <reason>`.
- Skip `*_harness.ts` alongside `*_test.ts` when indexing component
  `selector:` metadata (comment 4526017591).
- Only treat `first - 1` as covering a selector in `has_annotation`
  when it is a comment-only line (comment 4526017615).
- Enumerate files via `git ls-files -z` instead of `rglob` so untracked
  scratch files and local venvs are ignored (comment 4526017632).

Comment link: #1626 (comment)
Comment link: #1626 (comment)
Comment link: #1626 (comment)
Comment link: #1626 (comment)
Comment link: #1626 (comment)
Comment link: #1626 (comment)

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

Agreed — standardizing on data-testid is much cleaner and eliminates all of the CSS class and general attribute template regexes.

We split this into a 2-PR stack to keep each diff focused:

  1. feat(ui): migrate nightly verifier selectors to data-testid #1643 (feat/ui-selector-data-testid): Adds explicit data-testid="..." attributes across the 20 Angular templates exercised by the nightly UI verifiers (plus [attr.aria-pressed] on the velocity toggle pills and data-testid="work-unit-row" on the primary work-unit row) and migrates ui_css_selectors.py and ui_entities_validator.py to [data-testid="..."].
  2. feat(lint): add UI selector contract lint for nightly verifiers #1626 (this PR, now stacked on feat(ui): migrate nightly verifier selectors to data-testid #1643): Replaces the class/attribute template regex indexing in check_ui_selector_contract.py with a direct data-testid literal index (dom-testid) + custom component tag index (dom-tag), requires # ui-contract: <reason> for any non-Material class (dom-class), non-standard attribute (dom-attribute), or :has-text/has_text= copy match (copy), and incorporates all five inline review fixes (*_harness.ts exclusion, comment-only has_annotation line-above rule, git ls-files -z tracked-file enumeration, and PR description alignment).

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new static lint tool, check_ui_selector_contract.py, along with its corresponding unit tests and Bazel build targets, to ensure that nightly UI verifier selectors remain in sync with Angular templates. To support this contract, numerous data-testid attributes have been added across various Angular HTML templates, and the Playwright CSS selectors in ui_css_selectors.py and related test files have been updated to reference these stable test IDs instead of fragile CSS classes. The feedback recommends expanding the list of standard HTML attributes in the linter to prevent redundant annotations and refactoring the _is_known helper function to simplify its conditional logic.

Add data-testid="job-row" to the queue jobs table row in
queue_jobs_table.ng.html and update JOBS_TABLE_ROW_SEL and
assert_no_runtime_errors to use data-testid selectors directly.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from
feat/ui-selector-data-testid (#1643).

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Keep <tr mat-row *matRowDef=... data-testid="job-row"></tr> on a single
line to match cmd/ui Prettier printWidth formatting.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Keep <tr mat-row *matRowDef=... data-testid="job-row"></tr> on a single
line to match cmd/ui Prettier printWidth formatting.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Comment thread .githooks/pre-commit Resolved
Add data-testid attributes for the remaining 11 selector constants in
ui_css_selectors.py (resource-dashboard, wip-banner,
resource-type-option, smart-resource-finder, jobs-paginator,
dashboards-list-page, dashboards-page, dashboard-clear-filter-button,
chart-card, summary-box, admin-panel) so 100% of ui_css_selectors.py
constants use data-testid.

Comment link: #1626 (comment)

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from
feat/ui-selector-data-testid (#1643).

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
@VLukiantsev
VLukiantsev removed this pull request from stack #1629 November 12, 2026 13:11
@VLukiantsev
VLukiantsev changed the base branch from main to feat/ui-selector-data-testid November 12, 2026 13:11
< 10BC8 /div>
@VLukiantsev
VLukiantsev added this pull request to stack #1650 November 12, 2026 13:11
Place static data-testid="resource-type-option" before [value]="resType"
to satisfy @angular-eslint/template/attributes-order.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Place static data-testid="resource-type-option" before [value]="resType"
to satisfy @angular-eslint/template/attributes-order.

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
janzak-dev
janzak-dev previously approved these changes Nov 12, 2026
Ensure [data-testid="admin-panel"] is mounted in the DOM when
isLoading() or healthStatusError() is truthy on cold start so
navigate_to_route("/admin", sel.ADMIN_PANEL_ROOT_SEL, ...) reaches
wait_for_selector_with_retry.

Comment link: #1643 (comment)

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Step up one line in has_annotation when the line immediately above a
selector string ends with '(' so # ui-contract: comments above
parenthesized multi-line assignments or calls are recognized.

Comment link: #1626 (comment)

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from
feat/ui-selector-data-testid (#1643).

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
VLukiantsev added a commit that referenced this pull request Nov 12, 2026
Add data-testid attributes for the remaining 11 selector constants in
ui_css_selectors.py (resource-dashboard, wip-banner,
resource-type-option, smart-resource-finder, jobs-paginator,
dashboards-list-page, dashboards-page, dashboard-clear-filter-button,
chart-card, summary-box, admin-panel) so 100% of ui_css_selectors.py
constants use data-testid.

Comment link: #1626 (comment)

TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0