8000 `Binary` type, byte primitives, codecs, and binary I/O · Issue #138 · kexhq/kex · GitHub
[go: up one dir, main page]

Skip to content

Binary type, byte primitives, codecs, and binary I/O #138

Closed

Description

@akoskovacs

Kex needs a nominal type for arbitrary byte sequences. Byte is the
canonical spelling of UInt8; [Byte] remains the materialized list form,
but it is not a suitable storage representation for files, HTTP bodies,
digests, or large blobs.

This issue supersedes the earlier record Bytes { raw: String } proposal.
A public raw field would not enforce the byte/text boundary: callers could
run text operations on it directly or construct the record themselves.

Representation

Expose an opaque Kex type:

type Binary

Its BEAM value is {'Binary', Bin}, where Bin is a native Erlang binary.
The interpreter uses the equivalent opaque tagged value with byte-string
storage. The tag provides runtime identity and safe trait dispatch; the
payload remains a native binary so byte operations do not materialize lists.

Foreign bindings must explicitly unwrap on the way into Erlang and wrap on
the way back into Kex. Kex Binary is therefore distinct from a bare Erlang
binary even though both use binary storage.

Public API

module Binary do
  let fromBytes(values: [Byte]) -> Binary
  let fromHex(text: String) -> Binary?
  let fromBase64(text: String) -> Binary?
end

make Binary, implement: Showable, Inspectable do
  bytes    :> [Byte]
  length   :> Integer
  empty?   :> Bool
  at       :> Integer -> Byte?
  take     :> Integer -> Binary
  drop     :> Integer -> Binary
  +        :> Binary -> Binary
  hex      :> String
  base64   :> String
end

Boundary behavior follows the collection APIs:

  • at returns None for a negative or out-of-range index.
  • take(n <= 0) is empty; an oversized take returns the whole value.
  • drop(n <= 0) returns the whole value; an oversized drop is empty.
  • bytes is the explicit, allocating conversion to [Byte].

Conversions use the existing protocol:

text.to(Binary)   # Binary?, always Just for a String
data.to(String)   # String?, validates UTF-8

Keep the existing String.bytes : [Byte] and
String.fromBytes([Byte]) -> String? APIs for compatibility.

Showable and Inspectable must both render only
#Binary<N bytes>. They must never interpolate or decode the payload.

Hex is lowercase and strict: decoding rejects uppercase, odd length,
whitespace, prefixes, and non-hex input. Base64 uses the standard RFC 4648
alphabet with canonical padding; decoding rejects whitespace, URL-safe
characters, malformed or missing padding, and other noncanonical input.

BEAM primitives

The BEAM backend should operate directly on the wrapped binary:

  • byte_size/1 for length
  • binary:at/2 for at
  • binary:part/3 for take and drop, preserving sub-binaries
  • iolist_to_binary/1 for +
  • binary:encode_hex and native binary hex decoding
  • OTP's base64 module, with canonical-input validation

No Binary primitive may pass through a character list or Unicode operation.
The interpreter implementation must match the observable boundary behavior.

Byte invariant

Byte remains the canonical name for UInt8, but values typed as Byte
must actually be in 0..255 so Binary.fromBytes([Byte]) -> Binary is total:

  • Contextually typed integer literals are accepted only in range, including
    elements of a [Byte] literal.
  • Unchecked narrowing such as integer.as(Byte) is rejected.
  • Dynamic narrowing uses checked integer.to(Byte) -> Byte?.
  • Widening from Byte to Integer remains total.

The runtime constructor should still assert the invariant defensively, but a
failure is an internal contract violation rather than part of the public API.

Binary and text I/O

#139 established Readable and Writable. Complete those interfaces with
raw operations beside the text operations:

trait Readable do
  get       :> Result<String?, ReadError>
  getLine   :> Result<String?, ReadError>
  readLine  :> Result<String?, ReadError>
  read      :> Result<String, ReadError>
  readBytes :> Result<Binary, ReadError>
end

trait Writable do
  writeBytes :> Binary -> Void
end

type ReadError = ReadFailed | InvalidUtf8(Integer)

Ok(None) is EOF for character and line reads. Draining an exhausted source
returns Ok("") or Ok(Binary.fromBytes([])). InvalidUtf8(offset) reports
the byte offset relative to that operation. A failed operation consumes the
bytes it attempted to read and never substitutes U+FFFD. get must read one
complete Unicode scalar, not one byte.

writeBytes writes the payload directly and must never call showValue.
File handles and standard-stream handles implement both additions.

Add whole-file helpers:

FS.File.readBytes(path)          # Result<Binary, FileError>
FS.File.writeBytes(path, binary) # Bool

Change FS.File.read to Result<String, FileError>. Extend FileError with
path-carrying ReadFailed and InvalidUtf8(path, offset) variants and migrate
existing callers.

Chunked feeds and backpressure remain separate work; readBytes drains the
remaining input.

Digest

Preserve the existing hex API and add a raw overload:

Digest.sha256(String) -> String
Digest.sha256(Binary) -> Binary

Acceptance

Interpreter and BEAM tests must cover:

  • Empty values, embedded NULs, all 256 byte values, malformed UTF-8, and
    large payloads.
  • Indexing, clamped slicing, concatenation, equality, list conversion, and
    length-only rendering.
  • Strict canonical hex/base64 round trips and rejection cases.
  • Valid Byte literals and widening; rejection of negative/256 literals,
    [104, 300] as [Byte], and unchecked .as(Byte) narrowing; checked
    .to(Byte) boundaries.
  • Raw and hex SHA-256 against known vectors.
  • Real and mocked files, standard streams, EOF, UTF-8 errors with offsets,
    and lossless round trips of invalid text.
  • Compile-time enforcement that every Readable/Writable implementation
    supplies the new methods.
  • Generated Core calls the Binary intrinsic module and slicing uses
    binary:part, not binary_to_list.

Related

Activity

  1. changed the title [-]`Binary` type: bytes with hidden storage, `[Byte]` as its view[/-] [+]`Bytes` type and byte-level primitives (implements sql-types-plan §2)[/+] on Aug 19, 2026
  2. added this to the 0.4.0 milestone on Aug 26, 2026
  3. changed the title [-]`Bytes` type and byte-level primitives (implements sql-types-plan §2)[/-] [+]`Binary` type, byte primitives, codecs, and binary I/O[/+] on Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0