FFFF GitHub - tashfeenahmed/magictap: Touch input for a laptop screen with no touchscreen: your eyes pick the button, the tap sound says when. Built-in camera + mic only. · GitHub
[go: up one dir, main page]

Skip to content
tashfeenahmedPublic

About

Touch input for a laptop screen with no touchscreen: your eyes pick the button, the tap sound says when. Built-in camera + mic only.

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

Magic Tap

Touch input for a laptop screen that has no touchscreen.
Your eyes pick the button. The sound of your tap says when.
Only your laptop's built-in camera and microphone, with no extra hardware.

License: MIT Python 3.12 macOS 14+ Apple Silicon Extra hardware: none Tests Last commit

Quickstart · Integrate · How it works · Results · Troubleshooting


A person looks at one of six on-screen zones and taps the laptop screen on it: the camera tracks the gaze (where), the microphone hears the tap (when)

The camera follows your gaze to find where; the microphone hears the tap to know when.

Quickstart

uv tool install git+https://github.com/tashfeenahmed/magictap   # installs the `magictap` command
magictap setup      # checks your laptop, eye calibration (~6 min), tap training (~2 min), re-centre (15 s)
magictap buttons    # try it: look at a button and tap the screen on it

Each new session (different seat, lid angle or lighting), run magictap recenter first: 15 seconds, 6 dots. It removes the drift between sessions and is the single biggest accuracy boost (see Results).

Don't have uv? Install it with curl -LsSf https://astral.sh/uv/install.sh | sh (docs).

Every full-screen step waits for Space before it starts, and Esc quits. Models download automatically on first use (~30 MB, plus PyTorch). Your calibration and models are stored in ~/Library/Application Support/MagicTap (change it with MAGICTAP_HOME).

From source

git clone https://github.com/tashfeenahmed/magictap && cd magictap
uv sync
uv run magictap setup
uv run magictap recenter   # at the start of each session
uv run magictap buttons

Requirements

  • A Mac laptop with Apple Silicon and its built-in camera and microphone, macOS 14 or later (developed on an M4 Max). Optimized for laptops: it relies on the camera sitting just above the screen.
  • Python 3.12 (uv installs it for you)
  • Permissions for your terminal app, in System Settings > Privacy & Security:
    • Camera and Microphone: always needed
    • Accessibility: only for magictap mouse, which clicks for you

magictap doctor checks all of this and tells you what to fix.

Commands

Command What it does
magictap setup Guided first run: doctor, then calibrate, taps and recenter
magictap recenter Start of every session (15 s): look at and touch 6 dots; corrects drift from seating, lid angle and light
magictap doctor Checks camera, mic, permissions, models and calibration
magictap calibrate Eye calibration, plus touch rounds on a grid. The eye mapping is trained on all your past calibrations together
magictap taps Teaches it your screen taps vs. typing, talking and noise
magictap buttons Big on-screen buttons with right/wrong feedback. Guided mode keeps learning as you use it
magictap mouse Use it in any app: tap the screen and it clicks where you're looking
magictap serve Local WebSocket service for your app (see below)
magictap preview Camera view with the face and eye overlay, for checking framing
magictap evaluate-demo Offline accuracy on a recording from magictap record-demo (with and without re-centre)

Integrate it

Zero code: mouse mode

magictap mouse

Look at something and tap the screen on it, and Magic Tap clicks there. This works with any app whose buttons are big enough (see Results for what "big enough" means).

Your app: the local service

magictap serve      # ws://127.0.0.1:8765

Your app tells Magic Tap where its buttons are and gets an event when one is tapped. Coordinates are global screen points, with the origin at the top left of the main display.

Direction Message Meaning
app → Magic Tap {"type":"targets","targets":[{"id":"next","x":1200,"y":300,"w":400,"h":300}]} Your buttons (replaces the previous set)
app → Magic Tap {"type":"subscribe","gaze":true} Also stream the gaze point (~15 Hz)
Magic Tap → app {"type":"hello","version":"0.2.0","screen":[1728,1117]} Sent on connect
Magic Tap → app {"type":"tap","x":1395,"y":452,"target":"next","confidence":0.92,"t":…} A tap: where you were looking, which of your buttons that is (null if none) and how sure it is (0–1)
Magic Tap → app {"type":"ignored","reason":"not a tap"} A sound that was rejected: not a tap, unusual sound, keyboard/trackpad or eyes not on screen
Magic Tap → app {"type":"gaze","x":…,"y":…} Only if subscribed

JavaScript (clients/js/magictap.js, no dependencies):

import { connect, screenRect } from "./magictap.js";

const mt = connect();                       // reconnects automatically
mt.setTargets([{ id: "next", ...screenRect(document.querySelector("#next")) }]);
mt.on("tap", (tap) => { if (tap.target === "next" && tap.confidence > 0.6) nextSlide(); });

Python (magictap.client):

from magictap.client import MagicTapClient

client = MagicTapClient()
client.set_targets([{"id": "next", "x": 1200, "y": 300, "w": 400, "h": 300}])
client.on_tap = lambda tap: print(tap["target"], tap["confidence"])
client.run_forever()

Examples

  • examples/web/: a full-screen page with six buttons. Run magictap serve, then python3 -m http.server -d examples/web 8000 and open http://localhost:8000. Pages need to be served over http, because browsers block module imports from file://.
  • examples/python_client.py: reports left or right half taps.

Any language with WebSockets works the same way, including Swift's URLSessionWebSocketTask.

How it works

  • Where: your eyes.
    1. MediaPipe face mesh, with the face normalised for head pose.
    2. ETH-XGaze gaze estimation (ResNet-18).
    3. A per-user mapping to screen points, trained on all your calibration se 74C4 ssions together, so it isn't tied to one day's posture or light.
    4. A 15-second re-centre per session fits an affine correction for today's seating, lid angle and lighting.
    5. While you touch the screen your gaze reads slightly high; Magic Tap learns that shift and subtracts it.
  • When: the tap sound.
    1. 3 ms onset detection against an adaptive noise floor, with clips aligned to the exact sample of impact.
    2. A classifier trained on your taps vs. your everyday noises.
    3. An "unusual sound" check.
    4. Sounds within 150 ms of a key or trackpad press are ignored.
  • Fusion: a tap is placed where your eyes last settled (a fixation) in the 0.8 s before it, which skips the glance from the previous button. If fewer than 60% of those gaze samples agree on a button, Magic Tap says "not sure" instead of clicking.

CONFIGURATION.md has the exact settings and what we learned building it.

Results so far (one user, not a controlled study)

Right button on a 3×2 grid, new session, after a 6-dot re-centre 18 / 18 (100 %) on held-out presses
Same, without the re-centre 11 / 24 (46 %)
Gaze error on a session it never saw (3 sessions, leave-one-out, after re-centre) 148 pt median
Taps detected 91–95 %
False taps from typing, talking, knocks and music 0 of 73

How the 18/18 was measured: an offline evaluation of a recorded session from a different day than any calibration. The first press of each of the 6 cells stood in for magictap recenter, and the other 18 presses were scored with nothing from them used in fitting. Reproduce it with magictap evaluate-demo.

Caveats: it's a small sample from one person (the 95 % confidence interval for 18/18 starts at about 81 %), it's offline rather than live, and the presses were finger rests rather than heard taps. Live validation with magictap buttons after a re-centre is next. Magic Tap is built for a handful of large buttons, not precise pointing.

Troubleshooting

  • Nothing happens / it never says "ready": run magictap doctor. Camera and Microphone must be allowed for the app you run it from (Terminal, iTerm, VS Code…).
  • Wrong button: run magictap recenter (15 s); that fixes most of it. If it keeps happening, recalibrate in the position you actually use. Keep text and camera previews out of the middle of the screen while calibrating; anything readable pulls your eyes away from the dots.
  • Taps not heard, or heard when you didn't tap: run magictap taps again in the room you'll use it in. Tap with your fingertip; gentle taps work, but firm ones are more reliable.
  • Clicks in mouse mode don't happen: allow Accessibility for your terminal app.

Credits

  • Gaze model: ETH-XGaze ResNet-18 weights by hysts (ptgaze, MIT). The 3D face template in magictap/assets comes from the same project; its licence is included.
  • Face, pose and hand landmarks: MediaPipe.
  • Tap-detector ideas (short-window onsets, exact alignment, novelty rejection and learning from evaluation taps): JustinGamer191/Holo and its forks.

License

MIT, see LICENSE.

About

Touch input for a laptop screen with no touchscreen: your eyes pick the button, the tap sound says when. Built-in camera + mic only.

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0