Repository navigation
feat(lint): add UI selector contract lint for nightly verifiers - #1626
VLukiantsev wants to merge 29 commits into
Conversation
…ct lint The verify step in nightly-examples-e2e.yml stopped forwarding --expected_work_units after nightly-test-suite.yml was deleted (#1206), so verifiers silently fell back to hardcoded defaults. The ROD UI verifier and the simple_job duration tag were hardcoded outright. - Matrix entries launching examples/simple_job declare expected_work_units and expected_duration_sec; the verify step forwards them via env. - verifier_cli gains --expected_duration_sec; simple_job and ROD verifiers derive expectations from the resolvers. - New tools/nightly_tests/check_matrix_contract.py enforces that launch args equal the expected_* fields and that the verify step forwards them; wired into .githooks/pre-commit and lint.yml. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on check_matrix_contract.py: - A space-form contract flag followed by another `--` flag resolves to an empty value instead of consuming the next flag. - Blank and comment lines no longer end the verify step block. - The verify step start must be indented less than its `id:` line, so nested list items in run blocks are skipped. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
- `_VERIFY_STEP_ID_RE` accepts `- id: verify-example` as the first step key. - When the id line itself opens the step, it is the step start; no backward search that could match an outer list item. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Nightly UI verifiers reference Angular DOM through hand-copied selector strings with no build-graph edge to cmd/ui/app, so template changes (e.g. #1522) pass presubmit and break nightly. check_ui_selector_contract.py collects selectors from the nightly verifier sources via ast and requires every custom tag, class and attribute to exist in the Angular templates/components (Material/CDK prefixes exempt). Literal :has-text / has_text copy matches require a `# ui-contract: <reason>` annotation; existing matches are annotated. Wired into .githooks/pre-commit and lint.yml. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new static analysis tool, check_ui_selector_contract.py, along with its corresponding unit tests and Bazel configuration. The tool validates that nightly UI verifier selectors (Playwright CSS selectors) match tags, classes, and attributes declared in Angular templates, and enforces that any copy-matching selectors are annotated with a # ui-contract comment. The pre-commit hooks and documentation have been updated to integrate this check, and existing test files have been annotated accordingly. The review feedback focuses on improving the robustness of the linter by supporting single quotes in templates, handling list unpacking, supporting keyword arguments in selector calls, and guarding directory lookups with safety checks.
There was a problem hiding this comment.
Code Review
This pull request introduces a new static analysis linter, check_ui_selector_contract.py, along with associated unit tests, Bazel build targets, and pre-commit hooks. The linter ensures that nightly UI verifier CSS selectors remain synchronized with Angular templates, requiring explicit # ui-contract: annotations for text-based matches. The feedback suggests enhancing the linter's robustness by updating the regular expressions for static classes and class bindings to support both single and double quotes, preventing potential false positives.
Address review feedback on check_ui_selector_contract.py: - Static `class` and `[class]`/`[ngClass]` bindings accept single or double quotes via a matching-quote backreference. - Selector constants may use list as well as tuple unpacking. - `selector=` keyword arguments to locator/wait_for_selector are collected. - A missing cmd/ui/app directory is a handled error instead of reporting every selector as missing. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new linter tool, check_ui_selector_contract.py, along with its unit tests, to ensure that nightly UI verifier selectors remain synchronized with Angular templates. It checks for undeclared DOM tags, classes, and attributes, and enforces annotations for text-matching selectors. The pre-commit hooks, Bazel build files, and documentation have been updated accordingly, and existing UI tests have been annotated to comply with the new contract. The review feedback suggests improving the TypeScript selector regex to support backticks and removing the strict keyword from zip() to ensure compatibility with Python versions older than 3.10.
TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
The pre-commit regex and lint.yml path filter duplicated SELECTOR_SOURCES in two other syntaxes with no parity check, so a new or renamed selector source would silently skip the gate. The lint takes ~0.2s; run it on every commit and CI lint run instead, matching check_matrix_contract and check_file_naming. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Replace the hand-kept SELECTOR_SOURCES glob list, which failed open for new or renamed verifiers, with ast discovery over every *.py under tools/nightly_tests (hidden dirs and __pycache__ skipped; syntax errors fail the lint; EXCLUDED_SOURCES opt-out). Add an unknown-selector-constant rule: every reference to a ui_css_selectors name must be a collected *_SEL/*_SEL_TMPL constant, so the naming convention is enforced rather than assumed. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Nightly UI verifiers reference Angular DOM through hand-copied selector strings with no build-graph edge to cmd/ui/app, so template changes (e.g. #1522) pass presubmit and break nightly. check_ui_selector_contract.py collects selectors from the nightly verifier sources via ast and requires every custom tag, class and attribute to exist in the Angular templates/components (Material/CDK prefixes exempt). Literal :has-text / has_text copy matches require a `# ui-contract: <reason>` annotation; existing matches are annotated. Wired into .githooks/pre-commit and lint.yml. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on check_ui_selector_contract.py: - Static `class` and `[class]`/`[ngClass]` bindings accept single or double quotes via a matching-quote backreference. - Selector constants may use list as well as tuple unpacking. - `selector=` keyword arguments to locator/wait_for_selector are collected. - A missing cmd/ui/app directory is a handled error instead of reporting every selector as missing. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
The pre-commit regex and lint.yml path filter duplicated SELECTOR_SOURCES in two other syntaxes with no parity check, so a new or renamed selector source would silently skip the gate. The lint takes ~0.2s; run it on every commit and CI lint run instead, matching check_matrix_contract and check_file_naming. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Replace the hand-kept SELECTOR_SOURCES glob list, which failed open for new or renamed verifiers, with ast discovery over every *.py under tools/nightly_tests (hidden dirs and __pycache__ skipped; syntax errors fail the lint; EXCLUDED_SOURCES opt-out). Add an unknown-selector-constant rule: every reference to a ui_css_selectors name must be a collected *_SEL/*_SEL_TMPL constant, so the naming convention is enforced rather than assumed. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
c9ec2e2 to
2692b16
Compare
|
Design Question: Before investing further in maintaining template regexes: Because the index is global and regex-based rather than a real Angular/CSS parser, it only proves a token name exists somewhere in the UI (not on the target component), while adding substantial regex surface area to keep in sync with Angular template syntax. Would it be simpler and less fragile to standardize nightly verifier selectors on explicit |
Add explicit `data-testid` attributes to the 20 Angular component templates exercised by the Python nightly UI verifiers (plus `[attr.aria-pressed]` on the velocity period toggle pills) and update `ui_css_selectors.py` and `ui_entities_validator.py` to query `[data-testid="..."]` instead of styling classes. This decouples nightly E2E selectors from CSS class names and template formatting ahead of the `check_ui_selector_contract.py` lint gate in #1626. Part of bug 581241116 TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Address review feedback on #1626 ( stacked on #1643 ): - Replace CSS class and general attribute template regex indexing with a literal `data-testid="..."` index (`dom-testid`); non-Material classes and non-standard attributes in nightly selectors now require `# ui-contract: <reason>`. - Skip `*_harness.ts` alongside `*_test.ts` when indexing component `selector:` metadata (comment 4526017591). - Only treat `first - 1` as covering a selector in `has_annotation` when it is a comment-only line (comment 4526017615). - Enumerate files via `git ls-files -z` instead of `rglob` so untracked scratch files and local venvs are ignored (comment 4526017632). Comment link: #1626 (comment) Comment link: #1626 (comment) Comment link: #1626 (comment) Comment link: #1626 (comment) Comment link: #1626 (comment) Comment link: #1626 (comment) TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
|
Agreed — standardizing on We split this into a 2-PR stack to keep each diff focused:
|
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new static lint tool, check_ui_selector_contract.py, along with its corresponding unit tests and Bazel build targets, to ensure that nightly UI verifier selectors remain in sync with Angular templates. To support this contract, numerous data-testid attributes have been added across various Angular HTML templates, and the Playwright CSS selectors in ui_css_selectors.py and related test files have been updated to reference these stable test IDs instead of fragile CSS classes. The feedback recommends expanding the list of standard HTML attributes in the linter to prevent redundant annotations and refactoring the _is_known helper function to simplify its conditional logic.
Add data-testid="job-row" to the queue jobs table row in queue_jobs_table.ng.html and update JOBS_TABLE_ROW_SEL and assert_no_runtime_errors to use data-testid selectors directly. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from feat/ui-selector-data-testid (#1643). TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Keep <tr mat-row *matRowDef=... data-testid="job-row"></tr> on a single line to match cmd/ui Prettier printWidth formatting. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Keep <tr mat-row *matRowDef=... data-testid="job-row"></tr> on a single line to match cmd/ui Prettier printWidth formatting. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Add data-testid attributes for the remaining 11 selector constants in ui_css_selectors.py (resource-dashboard, wip-banner, resource-type-option, smart-resource-finder, jobs-paginator, dashboards-list-page, dashboards-page, dashboard-clear-filter-button, chart-card, summary-box, admin-panel) so 100% of ui_css_selectors.py constants use data-testid. Comment link: #1626 (comment) TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from feat/ui-selector-data-testid (#1643). TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Place static data-testid="resource-type-option" before [value]="resType" to satisfy @angular-eslint/template/attributes-order. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Place static data-testid="resource-type-option" before [value]="resType" to satisfy @angular-eslint/template/attributes-order. TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Ensure [data-testid="admin-panel"] is mounted in the DOM when
isLoading() or healthStatusError() is truthy on cold start so
navigate_to_route("/admin", sel.ADMIN_PANEL_ROOT_SEL, ...) reaches
wait_for_selector_with_retry.
Comment link: #1643 (comment)
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Step up one line in has_annotation when the line immediately above a
selector string ends with '(' so # ui-contract: comments above
parenthesized multi-line assignments or calls are recognized.
Comment link: #1626 (comment)
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Merge job-row and error-banner selector updates from feat/ui-selector-data-testid (#1643). TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Add data-testid attributes for the remaining 11 selector constants in ui_css_selectors.py (resource-dashboard, wip-banner, resource-type-option, smart-resource-finder, jobs-paginator, dashboards-list-page, dashboards-page, dashboard-clear-filter-button, chart-card, summary-box, admin-panel) so 100% of ui_css_selectors.py constants use data-testid. Comment link: #1626 (comment) TAG=agy CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024
Summary
Adds
tools/lint/check_ui_selector_contract.py, a pre-commit and CI lint that enforces thedata-testidselector contract between the Python nightly UI verifiers (tools/nightly_tests/) and the Angular templates (cmd/ui/app/**). A template change that removes adata-testidor component tag a verifier depends on, or a verifie 8000 r that couples to non-framework CSS classes, non-standard attributes, or user-facing copy without an explicit# ui-contract: <reason>annotation, now fails at commit time instead of in the 04:00 UTC nightly run.Motivation & Context
#1522 changed the ROD queue jobs table to render the experiment title, and the nightly ROD verifier matched rows with
has_text=f"Experiment #{xid}". Every presubmit passed because the verifier has no build-graph edge tocmd/ui/app/**, its unit test mocks the DOM, and the KinD E2E never visits/resources.Following the design review on #1626:
data-testidattributes (feat(ui): migrate nightly verifier selectors to data-testid #1643) avoids regex-indexingclass=,[class.x], and[ngClass]across Angular templates and decouples E2E verification from CSS styling refactors.:has-text/has_text=copy matches are disallowed unless accompanied by an explicit# ui-contract: <reason>annotation on the selector line or a comment-only line immediately above.Key Changes
tools/lint/check_ui_selector_contract.py(+ test, Bazel targets).git ls-files -z -- tools/nightly_testsandgit ls-files -z -- cmd/ui/app cmd/ui/index.html(matchingcheck_file_naming.py), so untracked scratch files and local venvs are ignored. Any syntax error in a tracked nightly file fails the lint, andEXCLUDED_SOURCESlets files opt out (empty today). Collects*_SEL/*_SEL_TMPLconstants (including tuple/list unpacking and annotated assignments) and the first positional orselector=argument of.locator()/.wait_for_selector().{...}placeholders are wildcards.dom-testid: every[data-testid="..."]value must match a literaldata-testid="..."attribute incmd/ui/app/**/*.ng.htmlorcmd/ui/index.html(bare[data-testid]without a literal value also fails).dom-tag: a custom element (with-) must be a declared Angular componentselector:incmd/ui/app/**/*.ts(skipping*_test.tsand*_harness.ts) or appear as a tag in a template.dom-class/dom-attribute: non-Material classes and non-standard attributes fail unless the selector carries# ui-contract: <reason>.mat-,mdc-,cdk-tags, classes, and attributes) and thehref,disabled,aria-*attributes.copy::has-text("…")andhas_text="…"literals need# ui-contract: <reason>on the same line or a comment-only line immediately above.unknown-selector-constant: everyalias.NAMEor imported name taken fromui_css_selectorsmust be a*_SEL/*_SEL_TMPLconstant collected from that module, enforcing the naming convention..githooks/pre-commitruns the lint on every commit aftercheck_file_naming(~0.2 s; no path gate, avoiding duplicate scope lists that could drift silently).lint.ymlgets an unconditional "Check Nightly UI Selector Contract" step.tools/README.mdindex entry; one line intools/nightly_tests/README.md.Assumptions
mat-,mdc-,cdk-) are exempt by prefix — they are generated by Angular Material at runtime and never appear as staticdata-testidattributes in templates — if wrong, a Material DOM change after a dependency bump is still caught only by nightly.:has-text(...)andhas_text=copy matches are linted — they are the selector-level pattern behind feat(rod): display experiment title in queue jobs table #1522 — if wrong,has_not_text=(one use today, an em-dash placeholder check) and post-hoctext_content()assertions are still caught only by nightly./resourcesE2E scenario,resolve_test_scope.pymapping, and proto/enum parity checks — separate slices of the same tracking bug — if wrong, coverage of those couplings stays nightly-only until they land.Verification & Testing
bazel test //tools/lint/..../.githooks/pre-commitpython3 tools/lint/check_ui_selector_contract.pyselector=keywords, implicit concatenation, f-strings, and placeholdersdom-testidmatches, missing testids, and bare[data-testid]without a literal value# ui-contract:*_test.tsand*_harness.tsexclusion from component selector indexing# ui-contract:on a preceding code line does not bleed into the next selector)git ls-files -ztracked-file enumeration (untracked broken files ignored; non-git--repo-roothandled)has_text=-only sources,EXCLUDED_SOURCES, andunknown-selector-constantvia module alias, relative import,import … as, andfrom … import (X, Y as Z)Related Issues / Bugs
Part of bug 581241116 (slice: UI selector contract lint). Stacked on #1643; follows #1624 (merged).
TAG=agy
CONV=14b9d4de-eba0-4b96-9ca0-760d4a2cc024