feat: add RFC 8941 Structured Fields parser and serializer by kawasima · Pull Request #138 · enkan/enkan

kawasima · 2026-04-03T01:25:56Z

Summary

Index-based recursive descent parser for RFC 8941 Structured Field Values — no regex, designed for hot-path performance
Supports all SF types: Integer, Decimal, String, Token, Byte Sequence, Boolean, Date (RFC 9651), Display String (RFC 9651)
Supports all SF containers: Item, Inner List, List, Dictionary, with Parameters
Standalone utility in enkan/util/sf/ — Headers class unchanged
Foundation for Issue Add Idempotency-Key middleware (RFC 9651) #131 (Idempotency-Key) and Issue Add HTTP Message Signatures support (RFC 9421) #135 (HTTP Message Signatures)

API

// Parse
SfItem item = StructuredFields.parseItem("\"key-123\";q=0.8");
SfDictionary dict = StructuredFields.parseDictionary("sig1=(\"@method\");keyid=\"k1\"");
SfList list = StructuredFields.parseList("1, 2, 3");

// Serialize
String header = StructuredFields.serializeItem(item);

Performance

charAt() / indexOf() / substring() based scanning — no regex compilation or matching
SfParameters.EMPTY singleton for parameter-less items
All value types are Java records (zero boxing overhead)
Parser is stateless — caching is the middleware's responsibility

closes #127

Test plan

71 new tests covering all SF types, parse, serialize, and round-trip
Error cases: malformed input, unterminated strings, trailing garbage
Real-world header examples: Signature-Input, Idempotency-Key, Priority
All 375 tests pass

🤖 Generated with Claude Code

Index-based recursive descent parser for Structured Field Values (RFC 8941 + RFC 9651 Date/Display String extensions). No regex used. Supports all SF types: Integer, Decimal, String, Token, Byte Sequence, Boolean, Date, Display String, plus Item, Inner List, List, Dictionary with Parameters. Standalone utility in enkan/util/sf/ — does not modify Headers class. Middleware uses StructuredFields.parseItem/parseList/parseDictionary directly, consistent with existing patterns (ETagUtils, etc.). closes #127 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Introduce sealed interface SfMember (permits SfItem, SfInnerList) to replace Object in SfList and SfDictionary - SfList.members() returns unmodifiable list - SfByteSequence: override equals/hashCode with Arrays.equals/hashCode - SfSerializer: preserve negative sign for -0.0 decimals - SfSerializer: use codePointAt() for surrogate-safe Display String encoding - SfSerializer.serializeItemOrInnerList: exhaustive switch instead of if-instanceof chain (compile-time completeness via sealed type) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- SfDictionary: defensive copy of members in compact constructor - SfParameters: defensive copy of map in compact constructor - SfInnerList: List.copyOf() for immutable items - SfParser.parseDisplayString: accumulate percent-encoded bytes in ByteArrayOutputStream and decode as UTF-8 (fixes multi-byte chars like é = %C3%A9) - SfSerializer.serializeString: reject characters outside VCHAR/SP range per RFC 8941 §4.1.6 - Add tests for all fixes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Change record field types from LinkedHashMap to Map, wrapping with Collections.unmodifiableMap in the constructor. Consistent with SfList.members() and SfInnerList.items() which are already immutable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds a standalone RFC 8941 Structured Field Values parser/serializer (plus RFC 9651 Date + Display String support) under enkan/util/sf/, with a comprehensive new test suite to validate parsing, serialization, and round-trips.

Changes:

Introduces SF data model types (SfItem, SfList, SfDictionary, SfInnerList, SfParameters, SfValue, etc.).
Implements an index-based recursive descent parser (SfParser) and serializer (SfSerializer) with a small public facade (StructuredFields).
Adds 70+ JUnit tests covering valid/invalid cases and round-trip behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
enkan-web/src/main/java/enkan/util/sf/StructuredFields.java	Public static API entry points for parse/serialize.
enkan-web/src/main/java/enkan/util/sf/SfParser.java	Core RFC 8941/9651 parser implementation.
enkan-web/src/main/java/enkan/util/sf/SfSerializer.java	Core serializer implementation.
enkan-web/src/main/java/enkan/util/sf/SfValue.java	SF bare value model (records).
enkan-web/src/main/java/enkan/util/sf/SfItem.java	Item container type.
enkan-web/src/main/java/enkan/util/sf/SfInnerList.java	Inner list container type.
enkan-web/src/main/java/enkan/util/sf/SfList.java	List container type.
enkan-web/src/main/java/enkan/util/sf/SfDictionary.java	Dictionary container type.
enkan-web/src/main/java/enkan/util/sf/SfParameters.java	Parameter container type + EMPTY singleton.
enkan-web/src/main/java/enkan/util/sf/SfMember.java	Sealed interface for list/dict members.
enkan-web/src/main/java/enkan/util/sf/SfParseException.java	Parse exception type.
enkan-web/src/test/java/enkan/util/sf/StructuredFieldsTest.java	New test suite for parsing/serialization/round-trip.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-03T01:41:26Z

+    SfList parseList() {
+        skipSP();
+        List<SfMember> members = new ArrayList<>();
+        while (pos < input.length()) {


parseList only skips SP (space) at the beginning. Per RFC 8941 the list can start with OWS (SP / HTAB), so a leading tab would currently fail to parse. Consider using skipOWS() here (and similarly for other top-level parse entry points).

Copilot · 2026-04-03T01:41:26Z

+    }
+
+    SfDictionary parseDictionary() {
+        skipSP();


parseDictionary only skips SP (space) at the beginning. RFC 8941 allows OWS (SP / HTAB) around separators; a leading HTAB would currently cause parsing to fail. Use skipOWS() for initial/terminal whitespace handling at the top-level.

Suggested change

skipSP();

skipOWS();

Copilot · 2026-04-03T01:41:26Z

+    SfItem parseItem() {
+        skipSP();
+        SfValue bareItem = parseBareItem();
+        SfParameters params = parseParameters();
+        skipSP();
+        return new SfItem(bareItem, params);
+    }


parseItem trims only spaces via skipSP(). RFC 8941 top-level values allow OWS (includes HTAB), so inputs with leading/trailing tabs will be rejected even though they are valid. Consider switching these to skipOWS() (and align ensureEmpty() accordingly).

Copilot · 2026-04-03T01:41:26Z

+    void ensureEmpty() {
+        skipSP();
+        if (pos < input.length()) {
+            throw fail("Unexpected trailing characters");
+        }
+    }


ensureEmpty() only skips SP (space) before checking for trailing characters. For RFC 8941 top-level types, trailing OWS may include HTAB, which would currently trigger an "Unexpected trailing characters" error for otherwise valid input.

Copilot · 2026-04-03T01:41:27Z

+                pendingBytes.write((hexVal(h1) << 4) | hexVal(h2));
+            } else {
+                // Flush accumulated percent-encoded bytes as UTF-8
+                if (pendingBytes.size() > 0) {
+                    sb.append(pendingBytes.toString(StandardCharsets.UTF_8));
+                    pendingBytes.reset();
+                }
+                if (c == '"') {
+                    return new SfDisplayString(sb.toString());
+                } else if (c < 0x20 || c > 0x7e) {
+                    throw fail("Invalid character in Display String: 0x" + Integer.toHexString(c));
+                } else {


Display String parsing flushes percent-decoded bytes using pendingBytes.toString(UTF_8), which will silently replace invalid UTF-8 sequences. RFC 9651 display strings are defined as percent-encoded UTF-8 bytes, so invalid sequences should be rejected (e.g., decode with a CharsetDecoder configured with CodingErrorAction.REPORT).

Copilot · 2026-04-03T01:41:27Z

+        Iterator<Map.Entry<String, SfMember>> it = dict.members().entrySet().iterator();
+        while (it.hasNext()) {
+            Map.Entry<String, SfMember> entry = it.next();
+            sb.append(entry.getKey());
+            SfMember member = entry.getValue();
+            if (member instanceof SfItem item && item.value() instanceof SfBoolean b && b.value()) {
+                serializeParameters(sb, item.parameters());
+            } else {


serializeDictionary writes dictionary keys directly (sb.append(entry.getKey())) without validating that they conform to RFC 8941 key syntax. This can produce invalid output and also allows CR/LF injection if untrusted strings are ever serialized into headers. Recommend validating keys (and rejecting CTL / non-key chars) before appending.

Copilot · 2026-04-03T01:41:27Z

+        switch (value) {
+            case SfInteger v -> sb.append(v.value());
+            case SfDecimal v -> serializeDecimal(sb, v.value());
+            case SfString v -> serializeString(sb, v.value());
+            case SfToken v -> sb.append(v.value());
+            case SfByteSequence v -> {


SfToken values are appended without validation. As with dictionary keys, this can generate invalid Structured Field output and may enable header injection if token contents include CTLs (e.g., CR/LF). Consider validating the token characters against the RFC 8941 token grammar during serialization (or at construction time).

Copilot · 2026-04-03T01:41:27Z

+    record SfByteSequence(byte[] value) implements SfValue {
+        @Override
+        public boolean equals(Object o) {
+            return o instanceof SfByteSequence other && Arrays.equals(value, other.value);
+        }
+
+        @Override
+        public int hashCode() {
+            return Arrays.hashCode(value);
+        }
+
+        @Override
+        public String toString() {
+            return "SfByteSequence[:" + Base64.getEncoder().encodeToString(value) + ":]";
+        }
+    }


SfByteSequence stores and exposes a mutable byte[] without defensive copies. This breaks immutability (callers can mutate the array after construction or via the accessor) and can violate the equals/hashCode contract if instances are used in hashed collections. Add a canonical constructor that copies the input array and have the accessor return a copy (or wrap in an immutable type).

Copilot · 2026-04-03T01:41:28Z

+public record SfList(List<SfMember> members) {
+
+    public SfList {
+        members = Collections.unmodifiableList(members);
+    }


SfList wraps the provided list with Collections.unmodifiableList but does not defensively copy it. If the caller mutates the original list after constructing SfList, the record’s contents will change. Use List.copyOf(members) to defensively copy (consistent with SfInnerList).

Copilot · 2026-04-03T01:41:28Z

+        void parseByteSequence() {
+            SfItem item = StructuredFields.parseItem(":cHJldHR5IG1hY2hpbmU=:");
+            assertThat(item.value()).isInstanceOf(SfByteSequence.class);
+            byte[] decoded = ((SfByteSequence) item.value()).value();
+            assertThat(new String(decoded)).isEqualTo("pretty machine");
+        }


This test converts decoded bytes to a String using the platform default charset (new String(decoded)), which can be non-UTF-8 and make the test environment-dependent. Prefer new String(decoded, StandardCharsets.UTF_8) (or US_ASCII for this vector).

- Parser: use skipOWS() (SP+HTAB) instead of skipSP() for top-level parse entry points and ensureEmpty(), per RFC 8941 OWS rules - Parser: DisplayString flushes percent-encoded bytes through a strict CharsetDecoder (CodingErrorAction.REPORT) to reject invalid UTF-8 - Serializer: validate dictionary keys and parameter keys against RFC 8941 key grammar before appending (prevents header injection) - Serializer: validate Token values against RFC 8941 token grammar - SfByteSequence: defensive clone() in constructor and accessor - SfList: use List.copyOf() for true defensive copy - Test: use StandardCharsets.UTF_8 for byte[] to String conversion Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Extract shared character helpers into SfChars to eliminate duplication between SfParser and SfSerializer - Serializer: validate Integer range (max 15 digits) and Decimal range (max 12-digit integer part) per RFC 8941 §3.3.1/§3.3.2 - StructuredFields: null checks on serialize methods - parseList/parseDictionary: allow empty input (returns empty container per RFC 8941 §4.2.1/§4.2.2) - Add 10 new edge case tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

kawasima and others added 4 commits April 3, 2026 10:25

kawasima requested a review from Copilot April 3, 2026 01:37

Copilot started reviewing on behalf of kawasima April 3, 2026 01:38

View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

kawasima and others added 2 commits April 3, 2026 10:45

kawasima merged commit 7ddda23 into develop Apr 3, 2026
1 check passed

Navigation Menu

Repository navigation

feat: add RFC 8941 Structured Fields parser and serializer - #138

feat: add RFC 8941 Structured Fields parser and serializer#138
kawasima merged 6 commits into
developfrom
feature/issue-127-structured-fields

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Summary

API

Performance

Test plan

Uh oh!

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants