Repository navigation
Conversation
… not on the list In a 2-level list the repeated node is both the list and its element, and its logical type describes the element. For a logical type that maps to an Arrow extension type (UUID, JSON, GEOMETRY, GEOGRAPHY, VARIANT, FILE), converting such a list went wrong in two ways: - a bare repeated field also had the extension put on the list, whose storage check rejects a List, so schema conversion failed; - a LIST group whose repeated child is the element converted, but the element lost its extension type. 3-level lists were unaffected. into_list now attaches the repeated node's extension type to the element field, as into_list_with_arrow_list_hint already does, and convert_field no longer adds it to the list.
Kurtiscwright
approved these changes
Nov 11, 2026
There was a problem hiding this comment.
Lgtm, was glad to see the three level list unit test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue filed; the bug is described below.
Rationale for this change
We have hit issues trying to place extension/annotation types onto elements of the legacy 2-level lists. The symptom is that the annotation gets placed on the resulting list wrapper, rather than the element contained in the list. When you then try to convert the Parquet schema to Arrow, arrow bails because it finds types in the wrong place. It will complain, for instance, that "UUID" cannot be placed on a List.
This affects every logical type that the converter maps to an Arrow extension type (extension.rs#L29-L93):
UUIDFIXED_LEN_BYTE_ARRAY(16)arrow.uuidarrow_canonical_extension_typesJSONBYTE_ARRAYarrow.jsonarrow_canonical_extension_typesGEOMETRY,GEOGRAPHYBYTE_ARRAYgeoarrow.wkbgeospatialVARIANTarrow.parquet.variantvariant_experimentalFILEarrow.parquet.filefile_experimentalOther logical types, such as
STRING,DATEorDECIMAL, map to a native Arrow data type and carry no extension type, so they are not affected.The two forms of 2-level list fail differently:
A bare repeated field gets the extension type attached to the list field, and
Field::try_with_extension_typerejects it because the extension'ssupports_data_typedoes not accept aList(field.rs#L578, uuid.rs#L55-L62):fails with
Uuid data type mismatch, expected FixedSizeBinary(16), found List(non-null FixedSizeBinary(16), field: 'col', metadata: {"ARROW:extension:name": "arrow.uuid"}). Each of the other types fails with its own storage check, for exampleJson data type mismatch, expected one of Utf8, LargeUtf8, Utf8View, found List(...).A LIST group whose repeated child is the element converts without error, but the element silently loses its extension type:
converts to
List<FixedSizeBinary(16)>with noarrow.uuidon the element.3-level lists are not affected: their element is a separate, non-repeated node, converted on its own (complex.rs#L626).
What changes are included in this PR?
convert_fieldno longer attaches a repeated node's extension type to the list it converts to. This fixes the bare repeated field.ParquetField::into_listtakes the repeated node'sTypeand attaches its extension type to the element field, asinto_list_with_arrow_list_hintalready does. This fixes the LIST group whose repeated child is the element.Are these changes tested?
Yes. New tests in
arrow::schema::testscover UUID in a bare repeated primitive, in a LIST group whose repeated child is the element, in a 3-level list, and in a list of 2-level lists, plus JSON in both 2-level forms. All but the plain 3-level test fail onmainand pass with this change.Are there any significant user-facing changes?
No API change.
Human generated, AI validated.