8000 fix(api): refuse a fields format on a type that cannot format, like ES by amartya1523 · Pull Request #2986 · xerj-org/xerj · GitHub
[go: up one dir, main page]
Skip to content

fix(api): refuse a fields format on a type that cannot format, like ES - #2986

Open
amartya1523 wants to merge 4 commits into
xerj-org:mainfrom
amartya1523:fix/fields-format-unsupported
Open

fix(api): refuse a fields format on a type that cannot format, like ES#2986
amartya1523 wants to merge 4 commits into
xerj-org:mainfrom
amartya1523:fix/fields-format-unsupported

Conversation

@amartya1523
@amartya1523 amartya1523 commented Nov 6, 2026
Copy link
Copy Markdown
Contributor

AI-assisted change, run and reviewed by @amartya1523.

What this changes, and why

A fields entry can carry a format ({"field": "date", "format": "yyyy/MM/dd"}). In ES, FieldFetcher#create builds a value fetcher for every mapped field the entry matches, passing the format. A type with only one rendering refuses it (MappedFieldType#checkNoFormat, SourceValueFetcher.identity). FieldFetcher wraps that with the entry, so the shard fails and the search is a 400 (search_phase_execution_exception, root cause illegal_argument_exception):

error fetching [keyword]: Field [keyword] of type [keyword] doesn't support formats.
error fetching [dkeyword] which matched [d*]: Field [dkeyword] of type [keyword] doesn't support formats.
error fetching [flattened.bar]: Field [flattened.bar] of type [flattened] doesn't support formats.

XERJ ignored the format and returned the value unformatted (apply_field_format hands back anything that doesn't parse as a date), with a 200. A caller asking for "format": "yyyy" on the wrong field couldn't tell. Three catch-lenient.txt lines are these cases (search/330_fetch_fields.yml):

  • error includes field name
  • error includes glob pattern
  • error for flattened includes whole path

Fix (es_compat.rs):

  • New fields_format_failure runs in search_impl after target resolution (beside the closed-index and allow_no_indices handling), so it reads each participating index's mapping. For every fields entry that has a string format:
    • Exact name: the mapped leaf, including a multi-field (name.raw), or flattened for a key below a flattened field (flat.some.key), which ES serves through the flattened keyed sub-type.
    • Wildcard: every mapped leaf the pattern matches, in path order, through the same glob_match_simple the search path uses. A match that can format (say date under d*) is passed over, and the first one that can't is reported with which matched [pattern].
    • A type in the new FORMATLESS_FIELD_TYPES fails: keyword, constant_keyword, wildcard, text, match_only_text, boolean, ip, version, the numeric types, token_count and flattened. Dates, geo and range types take a format, and any type not listed is left alone rather than guessed at.
  • New mapped_leaf_fields builds the full-path leaf list: multi-fields included, object / nested nodes excluded, no bare leaf aliases. The existing collect_leaf_types adds short names for dotted leaves, which would make a wildcard match fields it shouldn't.
  • The error uses the existing search_shard_validation_error envelope.

Deliberately not refused:

  • A name defined as a runtime field (in the body's runtime_mappings or the index's runtime section). The runtime type decides there, and this doesn't model it.
  • A search where some participating index can format the field. ES fails only the other shards and answers a partial 200, which this path doesn't model, so it keeps today's answer.

Evidence

New test module fields_format_tests (es_compat.rs), run against main first:

# main 83c9f0d12
test es_compat::fields_format_tests::a_format_on_a_formatless_field_is_a_400 ... FAILED
  left: 200
 right: 400
test es_compat::fields_format_tests::formattable_and_unmatched_fields_still_fetch ... ok

# this branch
test es_compat::fields_format_tests::a_format_on_a_formatless_field_is_a_400 ... ok
test es_compat::fields_format_tests::formattable_and_unmatched_fields_still_fetch ... ok
  • First test: dkeyword, d* (which also matches the date field, passed over), a long, a multi-field name.raw and a flattened key flat.bar. Each asserts the 400, the outer type, the root-cause type and ES's exact reason.
  • Second test:
    • date with yyyy/MM/dd still renders 1990/12/29
    • geo_point with wkt still answers
    • an unmapped name and a wildcard that matches nothing still answer
    • a runtime field named like a mapped long still answers
    • /test,other where other maps the field as a date still answers

Checks

  • cargo fmt -p xerj-api -- --check
  • cargo test --profile quick -p xerj-api --lib: 432 passed, 0 failed
  • YAML: the three existing cases now enforce it; their catch-lenient lines are deleted. The other fields + format cases in the suite (date in 330_fetch_fields.yml and 240_date_nanos.yml, geo_point geojson/wkt) are types that take a format. docvalue_fields (where numbers do take a #.0 format) isn't touched.
  • Docs: none describe the old behaviour.

Not run locally:

  • The ES-YAML suite, --release builds, clippy and the integration binaries. Disk on this machine is at 99%; CI runs all of these.

Provenance

  • Written by: AI-assisted, run and reviewed by @amartya1523
  • Verified: the test outputs above, on macOS 26 (aarch64), quick profile, based on main 83c9f0d12.
  • Assumed:
    • The list of format-less types follows ES's mappers. It's intentionally short of complete: a type missing from it keeps today's behaviour instead of risking a false 400.
    • ES iterates a wildcard's matches in its own order; with more than one format-less match, the field named here may differ from ES's.
    • _msearch doesn't go through search_impl, so it's not covered.

@amartya1523
fix(api): refuse a fields format on a type that cannot format, like ES
fb1dc4f 
ES builds a value fetcher for every field a `fields` entry matches,
with the entry's format. A type with one rendering (keyword, text,
numbers, boolean, ip, flattened, ...) throws, and FieldFetcher wraps it
with the entry: "error fetching [f]: Field [f] of type [t] doesn't
support formats.", plus "which matched [pattern]" for a wildcard. The
shard fails, so the search is a 400. XERJ ignored the format and
returned the value unformatted.

The check runs after target resolution against each index's mapping,
covering multi-fields and keys below a flattened field. Runtime fields
are left alone, and a search where some index can format the field
still answers, since ES would return a partial result there.

Retires the three search/330_fetch_fields.yml error cases.
@amartya1523
test(api): pin the fields format refusal
24116e3 
Exact names, a wildcard, a long, a multi-field and a flattened key are a 400 with ES's reason; dates, geo_point, unmapped names, empty wildcards, runtime fields and a mixed-index search still fetch.
@cla-bot cla-bot Bot added the cla-signed label Nov 6, 2026
@amartya1523
fix(api): use ? in fields_format_failure for clippy
4d1fc14
@amartya1523
test(es-compat-yaml): retire the now-passing Test date formatting entry
c2ed6db 
The fields format refusal also makes search/330_fetch_fields.yml's "Test date formatting" (a format on a keyword field, catch: bad_request) pass; CI's runner flagged the catch-lenient line as stale.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@amartya1523
0