Repository navigation
Description
Kex needs a nominal type for arbitrary byte sequences. Byte is the
canonical spelling of UInt8; [Byte] remains the materialized list form,
but it is not a suitable storage representation for files, HTTP bodies,
digests, or large blobs.
This issue supersedes the earlier record Bytes { raw: String } proposal.
A public raw field would not enforce the byte/text boundary: callers could
run text operations on it directly or construct the record themselves.
Representation
Expose an opaque Kex type:
type Binary
Its BEAM value is {'Binary', Bin}, where Bin is a native Erlang binary.
The interpreter uses the equivalent opaque tagged value with byte-string
storage. The tag provides runtime identity and safe trait dispatch; the
payload remains a native binary so byte operations do not materialize lists.
Foreign bindings must explicitly unwrap on the way into Erlang and wrap on
the way back into Kex. Kex Binary is therefore distinct from a bare Erlang
binary even though both use binary storage.
Public API
module Binary do
let fromBytes(values: [Byte]) -> Binary
let fromHex(text: String) -> Binary?
let fromBase64(text: String) -> Binary?
end
make Binary, implement: Showable, Inspectable do
bytes :> [Byte]
length :> Integer
empty? :> Bool
at :> Integer -> Byte?
take :> Integer -> Binary
drop :> Integer -> Binary
+ :> Binary -> Binary
hex :> String
base64 :> String
end
Boundary behavior follows the collection APIs:
atreturnsNonefor a negative or out-of-range index.take(n <= 0)is empty; an oversized take returns the whole value.drop(n <= 0)returns the whole value; an oversized drop is empty.bytesis the explicit, allocating conversion to[Byte].
Conversions use the existing protocol:
text.to(Binary) # Binary?, always Just for a String
data.to(String) # String?, validates UTF-8
Keep the existing String.bytes : [Byte] and
String.fromBytes([Byte]) -> String? APIs for compatibility.
Showable and Inspectable must both render only
#Binary<N bytes>. They must never interpolate or decode the payload.
Hex is lowercase and strict: decoding rejects uppercase, odd length,
whitespace, prefixes, and non-hex input. Base64 uses the standard RFC 4648
alphabet with canonical padding; decoding rejects whitespace, URL-safe
characters, malformed or missing padding, and other noncanonical input.
BEAM primitives
The BEAM backend should operate directly on the wrapped binary:
byte_size/1forlengthbinary:at/2foratbinary:part/3fortakeanddrop, preserving sub-binariesiolist_to_binary/1for+binary:encode_hexand native binary hex decoding- OTP's
base64module, with canonical-input validation
No Binary primitive may pass through a character list or Unicode operation.
The interpreter implementation must match the observable boundary behavior.
Byte invariant
Byte remains the canonical name for UInt8, but values typed as Byte
must actually be in 0..255 so Binary.fromBytes([Byte]) -> Binary is total:
- Contextually typed integer literals are accepted only in range, including
elements of a[Byte]literal. - Unchecked narrowing such as
integer.as(Byte)is rejected. - Dynamic narrowing uses checked
integer.to(Byte) -> Byte?. - Widening from
BytetoIntegerremains total.
The runtime constructor should still assert the invariant defensively, but a
failure is an internal contract violation rather than part of the public API.
Binary and text I/O
#139 established Readable and Writable. Complete those interfaces with
raw operations beside the text operations:
trait Readable do
get :> Result<String?, ReadError>
getLine :> Result<String?, ReadError>
readLine :> Result<String?, ReadError>
read :> Result<String, ReadError>
readBytes :> Result<Binary, ReadError>
end
trait Writable do
writeBytes :> Binary -> Void
end
type ReadError = ReadFailed | InvalidUtf8(Integer)
Ok(None) is EOF for character and line reads. Draining an exhausted source
returns Ok("") or Ok(Binary.fromBytes([])). InvalidUtf8(offset) reports
the byte offset relative to that operation. A failed operation consumes the
bytes it attempted to read and never substitutes U+FFFD. get must read one
complete Unicode scalar, not one byte.
writeBytes writes the payload directly and must never call showValue.
File handles and standard-stream handles implement both additions.
Add whole-file helpers:
FS.File.readBytes(path) # Result<Binary, FileError>
FS.File.writeBytes(path, binary) # Bool
Change FS.File.read to Result<String, FileError>. Extend FileError with
path-carrying ReadFailed and InvalidUtf8(path, offset) variants and migrate
existing callers.
Chunked feeds and backpressure remain separate work; readBytes drains the
remaining input.
Digest
Preserve the existing hex API and add a raw overload:
Digest.sha256(String) -> String
Digest.sha256(Binary) -> Binary
Acceptance
Interpreter and BEAM tests must cover:
- Empty values, embedded NULs, all 256 byte values, malformed UTF-8, and
large payloads. - Indexing, clamped slicing, concatenation, equality, list conversion, and
length-only rendering. - Strict canonical hex/base64 round trips and rejection cases.
- Valid Byte literals and widening; rejection of negative/256 literals,
[104, 300]as[Byte], and unchecked.as(Byte)narrowing; checked
.to(Byte)boundaries. - Raw and hex SHA-256 against known vectors.
- Real and mocked files, standard streams, EOF, UTF-8 errors with offsets,
and lossless round trips of invalid text. - Compile-time enforcement that every
Readable/Writableimplementation
supplies the new methods. - Generated Core calls the Binary intrinsic module and slicing uses
binary:part, notbinary_to_list.
Related
- Extract
Readable/Writabletraits; moveIOonto handles #139 —Readable/Writableand handles as values - Net module —
Http→Net.Http, File-style interface #81 —Net.HTTP, whose response bodies should useBinary
Activity
[-]`Binary` type: bytes with hidden storage, `[Byte]` as its view[/-][+]`Bytes` type and byte-level primitives (implements sql-types-plan §2)[/+][-]`Bytes` type and byte-level primitives (implements sql-types-plan §2)[/-][+]`Binary` type, byte primitives, codecs, and binary I/O[/+]