8000 Apple camera preview: thicket-camera-preview-apple over AVCaptureSession (#148) by rleibman · Pull Request #158 · rleibman/thicket · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .scalafmt.conf
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ project.excludeFilters = [
"modules/motion-ios/src/main/scala/thicket/motion/ios/MotionShim.scala"
"modules/pickers-apple/src/main/scala/thicket/pickers/apple/PickersShim.scala"
"modules/camera-apple/src/main/scala/thicket/camera/apple/CameraShim.scala"
"modules/camera-preview-apple/src/main/scala/thicket/camera/preview/apple/PreviewShim.scala"
]

maxColumn = 120
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ modules/
pickers/ photo and document pickers, save dialog; pickers-gtk/, pickers-android/
camera/ taking a photo with the camera app; camera-android/, camera-apple/ (GTK, macOS: Camera.unavailable)
share/ the share sheet; share-android/, share-apple/ (GTK: Share.unavailable)
camera-preview/ a live camera preview in the app's layout; camera-preview-android/ (GTK: CameraPreview.unavailable)
camera-preview/ a live camera preview in the app's layout; camera-preview-android/, camera-preview-apple/ (GTK: CameraPreview.unavailable)
renderer-gtk/ GTK4 (Scala Native)
renderer-android/ android.view (Scala on ART)
renderer-apple/ AppKit + UIKit (Scala Native + Swift shim)
Expand Down
2 changes: 1 addition & 1 deletion bin/release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# The route the first release took (0.2.0, 2026-11-09; docs/14 §14.5):
#
# 1. The Mac clones the tag `v<version>` and runs `publish; platforms/publish` unsigned into
# `target/sona-staging`. Only a Mac can build every artefact: seven Apple modules carry
# `target/sona-staging`. Only a Mac can build every artefact: eight Apple modules carry
# Swift libraries that need Xcode, and off a Mac their `publish` is skipped. The Mac needs
# no gpg and no Central token.
# 2. The staging directory comes back here, without macOS's AppleDouble `._*` files.
Expand Down
2 changes: 2 additions & 0 deletions bin/verify-apple.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
# shim-motion compile thicket-motion-ios's Swift library, simulator only (#120)
# shim-pickers compile thicket-pickers-apple's Swift library, macOS and simulator (#125)
# shim-camera compile thicket-camera-apple's Swift library, simulator only (#129)
# shim-preview compile thicket-camera-preview-apple's Swift library, macOS and simulator (#148)
# abi shim-gen's consistency checks: ABI, C header, Scala externs, both shims
# secrets thicket-secrets-apple's tests against the real Keychain (#108): conformance, isolation
# link sbt nativeLink for the six Apple apps, after deleting their old outputs
Expand Down Expand Up @@ -190,6 +191,7 @@ run shim-device sh -c 'modules/device-apple/shim/build-shim.sh macos && modules/
run shim-motion modules/motion-ios/shim/build-shim.sh ios-sim
run shim-pickers sh -c 'modules/pickers-apple/shim/build-shim.sh macos && modules/pickers-apple/shim/build-shim.sh ios-sim'
run shim-camera modules/camera-apple/shim/build-shim.sh ios-sim
run shim-preview sh -c 'modules/camera-preview-apple/shim/build-shim.sh macos && modules/camera-preview-apple/shim/build-shim.sh ios-sim'
run abi sbt --error shimGen/testFull
run secrets sbt --error secretsApple/testFull
run link link_all
Expand Down
49 changes: 47 additions & 2 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,42 @@ lazy val cameraPreviewAndroid = project
.dependsOn(cameraPreview.jvm, rendererAndroid)
.settings(commonSettings, name := "thicket-camera-preview-android", description := "Thicket: a live camera preview on Android through Camera2", scalacOptions ++= Seq("-release", "17"))

/** The camera preview on iOS and macOS (#148): `AVCaptureSession` with an `AVCaptureVideoPreviewLayer` in a `UIView` or
* an `NSView`, and an `AVCapturePhotoOutput` on the same session for the capture. Through a Swift library of its own
* (`shim/`, `libthicketpreview.a`, both targets) carried in the jar like `pickersApple`'s, so an app that shows no
* preview links none of it. The permission is `deviceApple`'s. In `platforms`; building the Swift needs Xcode.
*/
lazy val cameraPreviewApple = project
.in(file("modules/camera-preview-apple"))
.enablePlugins(AutomateHeaderPlugin, com.github.sbt.git.GitVersioning, ScalaNativePlugin)
.dependsOn(cameraPreview.native, deviceApple, rendererApple, core.native)
.settings(
commonSettings,
name := "thicket-camera-preview-apple",
description := "Thicket: a live camera preview on iOS and macOS through AVCaptureSession"
)
// Generated by shimGen and compared byte for byte, like the renderer's `Shim.scala`.
.settings(headerSources / excludeFilter := HiddenFileFilter || "PreviewShim.scala")
.settings(
publish / skip := !isMac,
Compile / packageBin / mappings ++= {
val shim = (ThisBuild / baseDirectory).value / "modules" / "camera-preview-apple" / "shim"
val conv = fileConverter.value
if !isMac then Seq.empty
else {
for target <- Seq("macos", "ios-sim") do {
val code = scala.sys.process.Process(Seq("bash", (shim / "build-shim.sh").getAbsolutePath, target), shim).!
if code != 0 then sys.error(s"camera-preview-apple build-shim.sh $target failed with exit $code")
}
Seq(
shim / "build" / "libthicketpreview.a" -> "thicket-camera-preview-apple/macos/libthicketpreview.a",
shim / "build-ios" / "libthicketpreview.a" -> "thicket-camera-preview-apple/ios-sim/libthicketpreview.a",
shim / "include" / "thicket_preview.h" -> "thicket-camera-preview-apple/include/thicket_preview.h"
).map((f, path) => conv.toVirtualFile(f.toPath) -> path)
}
}
)

/** The camera on Android: the camera app's `ACTION_IMAGE_CAPTURE`, writing to a MediaStore entry. */
lazy val cameraAndroid = project
.in(file("modules/camera-android"))
Expand Down Expand Up @@ -1115,7 +1151,7 @@ lazy val galleryMacos = project
.in(file("examples/gallery/macos"))
.enablePlugins(AutomateHeaderPlugin, com.github.sbt.git.GitVersioning, BuildInfoPlugin, ScalaNativePlugin)
.settings(thicketBuildInfo("example.gallery.macos.buildinfo"))
.dependsOn(rendererApple, galleryShared.native, storageApple, secretsApple, deviceApple, locationApple, clipboardApple, hapticsApple, shareApple, pickersApple)
.dependsOn(rendererApple, galleryShared.native, storageApple, secretsApple, deviceApple, locationApple, clipboardApple, hapticsApple, shareApple, pickersApple, cameraPreviewApple)
.settings(commonSettings)
.settings(appleNativeSettings)
.settings(galleryAppleSharedSources)
Expand All @@ -1128,6 +1164,14 @@ lazy val galleryMacos = project
.settings(
nativeConfig ~= thicket.sbt.ThicketNativeFlags.applePickers(file("modules") / "pickers-apple" / "shim" / "build")
)
// thicket-camera-preview-apple's Swift library (#148): `modules/camera-preview-apple/shim/build-shim.sh` builds it.
// The executable runs outside an app bundle, so its NSCameraUsageDescription is an Info.plist embedded in it.
.settings(
nativeConfig ~= thicket.sbt.ThicketNativeFlags.appleCameraPreview(
file("modules") / "camera-preview-apple" / "shim" / "build"
),
nativeConfig ~= thicket.sbt.ThicketNativeFlags.appleInfoPlist(file("examples") / "gallery" / "macos" / "Info.plist")
)
.settings(
name := "gallery-macos",
publish / skip := true,
Expand All @@ -1138,7 +1182,7 @@ lazy val galleryIos = project
.in(file("examples/gallery/ios"))
.enablePlugins(AutomateHeaderPlugin, com.github.sbt.git.GitVersioning, BuildInfoPlugin, ScalaNativePlugin)
.settings(thicketBuildInfo("example.gallery.ios.buildinfo"))
.dependsOn(rendererApple, galleryShared.native, storageApple, secretsApple, deviceApple, locationApple, motionIos, cameraApple, clipboardApple, hapticsApple, shareApple, pickersApple)
.dependsOn(rendererApple, galleryShared.native, storageApple, secretsApple, deviceApple, locationApple, motionIos, cameraApple, clipboardApple, hapticsApple, shareApple, pickersApple, cameraPreviewApple)
.settings(commonSettings)
.settings(iosNativeSettings)
.settings(galleryAppleSharedSources)
Expand Down Expand Up @@ -1321,6 +1365,7 @@ lazy val platforms = project
cameraAndroid,
cameraApple,
cameraPreviewAndroid,
cameraPreviewApple,
shareAndroid,
clipboardApple,
hapticsApple,
Expand Down
6 changes: 5 additions & 1 deletion docs/12-component-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ every remaining entry before it is built, not after.
| Haptics | `modules/haptics`, `haptics-android`, `haptics-apple` | done on Android (2026-10-30) — `performHapticFeedback`, which `build.sh` finds in the system's vibrator history; GTK uses `Haptics.none`. Apple (#123, 2026-10-30): iOS's feedback generators and macOS's `NSHapticFeedbackManager` (a Force Touch trackpad); the self-test can only show the calls return — the simulator plays nothing. `modules/haptics/README.md` |
| Pickers | `modules/pickers`, `pickers-gtk`, `pickers-android`, `pickers-apple` | done on GTK (`GtkFileDialog`) and Android (the photo picker, the Storage Access Framework) (2026-10-30) — `Photos.pick(limit)`, `Documents.open(types)` and `save(name, bytes)`; every picked file is copied into the app's cache and handed back as a `Path`, by callback. The gallery's self-tests answer the real pickers: `build.sh` taps Android's, the GTK test answers the in-process chooser. Apple (2026-11-02, #125): `thicket-pickers-apple`, its own Swift library — `PHPickerViewController` and `UIDocumentPickerViewController` on iOS, tapped in the iOS gallery by the XCUITest tapper; `NSOpenPanel` and `NSSavePanel` sheets on macOS, whose OK cannot be pressed from the app on macOS 27 (the panels are out of process), so the self-test answers an accept through the panel's own completion with a known file and cancels with the panel's own `cancel(_:)`. `modules/pickers/README.md` |
| Camera capture | `modules/camera`, `camera-android` | done on Android 10+ (2026-10-30) — `Camera.capturePhoto`: the camera app's screen, the photo copied into the app's cache as a JPEG, the MediaStore entry it was written to deleted; GTK uses `Camera.unavailable`. The gallery's self-test presses the emulator camera's shutter and Done, and backs out. iOS (2026-11-02, #129): `thicket-camera-apple` over `UIImagePickerController` (its own Swift library), the photo written as a JPEG and copied into the cache; the app's `Info.plist` must declare `NSCameraUsageDescription` (else `IllegalStateException`). The iOS 27 simulator has a camera screen whose shutter takes nothing: the gallery opens it, checks one request at a time, and closes it with its X; the capture itself is unmeasured (no device). macOS uses `Camera.unavailable`. `modules/camera/README.md` |
| Camera preview | `modules/camera-preview`, `camera-preview-android` | done on Android 8+ (2026-11-03, `docs/17` step 2) — `CameraPreview`: Camera2 into a `TextureView` through the embedding point (`dsl.Platform`), `capture` a JPEG from the same session, `facing`, `running`; runs only while mounted and in front. The gallery's self-test runs the emulator's back camera: frames, a capture of the session's size, Home closes it, front reopens it, no front camera says so. GTK uses `CameraPreview.unavailable` (§17.8.2: later). Apple is #148. `modules/camera-preview/README.md` |
| Camera preview | `modules/camera-preview`, `camera-preview-android` | done on Android 8+ (2026-11-03, `docs/17` step 2) — `CameraPreview`: Camera2 into a `TextureView` through the embedding point (`dsl.Platform`), `capture` a JPEG from the same session, `facing`, `running`; runs only while mounted and in front. The gallery's self-test runs the emulator's back camera: frames, a capture of the session's size, Home closes it, front reopens it, no front camera says so. GTK uses `CameraPreview.unavailable` (§17.8.2: later). **Apple** (2026-11-03, #148, `camera-preview-apple`): `AVCaptureSession` with an `AVCaptureVideoPreviewLayer` in a `UIView` / `NSView`, `AVCapturePhotoOutput` for `capture`; built and its unavailable and not-permitted paths self-tested on both, but **no session has run yet**: the iOS 27 simulator lists no `AVCaptureDevice`, and the Mac's camera was not permitted (`docs/decisions.md`, 2026-11-03). `modules/camera-preview/README.md` |
| Share | `modules/share`, `share-android`, `share-apple` | done on Android (2026-11-01) — `Share.text` and `Share.file` through the chooser; a file is copied and served by Thicket's own `ThicketFileProvider`, read-only, with a grant for the chosen app; the app declares the provider in its manifest. GTK uses `Share.unavailable`. The gallery's self-test shares to its own share target, tapped in the real chooser. Done on macOS and iOS (#134) — `NSSharingServicePicker` and `UIActivityViewController`, a C helper, no Swift; the subject through a `UIActivityItemSource` made at run time; on iOS an XCUITest taps "Copy" in the sheet and `simctl pbpaste` reads the text; no macOS service is driven. `modules/share/README.md` |
| UI-thread seam | `UiThread`, `Host.install` | done — every host installs it since 2026-10-20; before that none did, so a ZIO effect's result was written from a worker thread. The default now throws |
| Apple ABI description + consistency check | `tools/shim-gen` | done — every `sui_*` function (110 externs in `Shim.scala`) and all 26 kind codes described; `thicket_apple.h` and `Shim.scala` are generated from it (adopted 2026-09-29, #3) and checked byte-for-byte; the two Swift shims' signatures and per-platform widget choice are checked against it, on any machine. Since #90 also `thicket-http-apple`'s 12 `thk_http_*` functions (`HttpAbi`): `thicket_http.h` and `HttpShim.scala` generated, `HttpShim.swift` checked |
Expand Down Expand Up @@ -418,6 +418,10 @@ Each measured on its own branch; the gallery with both sections has not been run
**231** with the screen locked, the RadioGroup and Stepper section's 13 included on each.
2026-11-03 (#145): macOS **228** (224 with the screen locked), iOS simulator ("iPhone 17e")
**239**, each with the Platform section's 6, its heading among them.
2026-11-03 (#148, before main's RadioGroup and Stepper): macOS **230**, iOS simulator ("iPhone 18
Pro") **245**, each with the Camera preview section's 6, its heading among them (unmounted, the
fits, the aspect, both facings; no session could run on either, so the live checks were not
reached).

Four rules the migration established:

Expand Down
7 changes: 4 additions & 3 deletions docs/14-releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ The artefacts, with explicit suffixes, because sbt 2 has no `%%%`:
| `net.leibman:thicket-camera-apple_native0.5_3` | taking a photo on iOS with `UIImagePickerController`, with its own Swift library inside the jar (`ios-sim/libthicketcamera.a`); the app's `Info.plist` needs `NSCameraUsageDescription`; on macOS, `Camera.unavailable` from `thicket-camera` |
| `net.leibman:thicket-share_3` / `_native0.5_3`, `thicket-share-android_3` | the share sheet; on Android the app declares `ThicketFileProvider`; on GTK, `Share.unavailable` |
| `net.leibman:thicket-camera-preview_3` / `_native0.5_3`, `thicket-camera-preview-android_3` | a live camera preview in the app's layout (`docs/17`): Camera2 on Android, the app declaring `CAMERA`; elsewhere, `CameraPreview.unavailable` from `thicket-camera-preview` |
| `net.leibman:thicket-camera-preview-apple_native0.5_3` | the preview on iOS and macOS: `AVCaptureSession` in a `UIView` / `NSView`, with its own Swift library inside the jar (`macos/` and `ios-sim/libthicketpreview.a`); the app's `Info.plist` needs `NSCameraUsageDescription` (a macOS executable outside a bundle embeds one, `ThicketNativeFlags.appleInfoPlist`) |
| `net.leibman:thicket-renderer-android_3` | the Android renderer (plain JVM bytecode, `-release 17`; R8 dexes it) |
| `net.leibman:thicket-renderer-apple_native0.5_3` | the AppKit and UIKit renderers, with the Swift shim inside the jar |
| `net.leibman:thicket-storage-apple_native0.5_3` | app directories and preferences on macOS and iOS (`thicket-storage` comes with it); no Swift |
Expand Down Expand Up @@ -165,7 +166,7 @@ person can measure that; this checks the half that does not need one.
The group id is `net.leibman` (decision log, 2026-10-20), on the Central Portal.

**The Mac builds; Linux signs and uploads** (decision log, 2026-11-12). Only a Mac can build
every artefact: seven Apple modules carry Swift libraries that need Xcode, and off a Mac their
every artefact: eight Apple modules carry Swift libraries that need Xcode, and off a Mac their
`publish` is skipped rather than shipping a jar with no library in it. The signing key and the
Central token stay on Linux, so the Mac needs neither gpg nor credentials.
`bin/release.sh` does the whole of it.
Expand Down Expand Up @@ -253,5 +254,5 @@ renderers and the platform modules that are not in `root`: `thicket-http-apple`,
`thicket-location-android`, `thicket-location-apple`, `thicket-motion-android`, `thicket-motion-ios`, `thicket-clipboard-gtk`,
`thicket-clipboard-android`, `thicket-clipboard-apple`, `thicket-haptics-android`,
`thicket-haptics-apple`, `thicket-pickers-gtk`, `thicket-pickers-android`, `thicket-pickers-apple`,
`thicket-camera-android`, `thicket-camera-apple`, `thicket-share-android`, `thicket-share-apple` and
`thicket-camera-preview-android`.
`thicket-camera-android`, `thicket-camera-apple`, `thicket-share-android`, `thicket-share-apple`,
`thicket-camera-preview-android` and `thicket-camera-preview-apple`.
3 changes: 2 additions & 1 deletion docs/16-device-services.md
Original file line number Diff line number Diff line change
Expand Up @@ -487,4 +487,5 @@ Every service gets a gallery section and self-test checks on every host where it
6. **Tier 3** as separate designs, live camera preview first if wanted. *The owner wants tier 3
(2026-11-01); the camera preview's design is `docs/17-camera-preview.md`, approved with every
recommendation (2026-11-01). The embedding point is built (2026-11-02, #146), and the preview
on Android (2026-11-03, `thicket-camera-preview`, `-android`); Apple is the Mac's, GTK later.*
on Android (2026-11-03, `thicket-camera-preview`, `-android`), and on Apple (2026-11-03, #148,
`-apple`; no session measured yet, `docs/17`); GTK later.*
6 changes: 4 additions & 2 deletions docs/17-camera-preview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Status: **approved, 2026-11-01; step 1 of §17.7, the embedding point, built on GTK and
Android 2026-11-02** (`WidgetKind.Platform`, `Prop.View`, `GtkPlatformView`,
`AndroidPlatformView`), and on AppKit and UIKit 2026-11-03 (`ApplePlatformView`, #145). Step 2, **the preview on Android, built 2026-11-03** (`thicket-camera-preview`,
`-android`; `modules/camera-preview/README.md`). Apple (step 3) is the Mac's; GTK (step 4) later. `docs/16` §16.3 puts the live camera
`-android`; `modules/camera-preview/README.md`). Step 3, **Apple, built 2026-11-03** (`-apple`, #148): no session
has run yet, since the iOS 27 simulator lists no capture device and the Mac's camera was not permitted (decision log). GTK (step 4) later. `docs/16` §16.3 puts the live camera
preview first in tier 3 ("each its own design"), and the owner said yes to tier 3 on
2026-11-01. This is that design. The owner answered §17.8's questions the same day, taking
every recommendation (decision log, 2026-11-01).
Expand Down Expand Up @@ -103,7 +104,8 @@ The largest piece in the plan (`docs/16` §16.3 says so). Proposed order, each i
Apple as a Mac issue. *Done on GTK and Android, 2026-11-02, and on Apple, 2026-11-03 (#145)
(decision log).*
2. `thicket-camera-preview` and `-android`, measured on the emulator. *Done, 2026-11-03.*
3. `-apple`, the Mac's (iOS and macOS).
3. `-apple`, the Mac's (iOS and macOS). *Built, 2026-11-03 (#148); the live preview not yet
measured on either (decision log).*
4. `-gtk`, after Android and Apple (§17.8.2).

## 17.8 Questions for the owner — answered 2026-11-01
Expand Down
Loading
Loading
0