Touch input for a laptop screen that has no touchscreen.
Your eyes pick the button. The sound of your tap says when.
Only your laptop's built-in camera and microphone, with no extra hardware.
Quickstart · Integrate · How it works · Results · Troubleshooting
The camera follows your gaze to find where; the microphone hears the tap to know when.
uv tool install git+https://github.com/tashfeenahmed/magictap # installs the `magictap` command
magictap setup # checks your laptop, eye calibration (~6 min), tap training (~2 min), re-centre (15 s)
magictap buttons # try it: look at a button and tap the screen on itEach new session (different seat, lid angle or lighting), run magictap recenter
first: 15 seconds, 6 dots. It removes the drift between sessions and is the single
biggest accuracy boost (see Results).
Don't have uv? Install it with curl -LsSf https://astral.sh/uv/install.sh | sh
(docs).
Every full-screen step waits for Space before it starts, and Esc quits.
Models download automatically on first use (~30 MB, plus PyTorch). Your
calibration and models are stored in ~/Library/Application Support/MagicTap
(change it with MAGICTAP_HOME).
git clone https://github.com/tashfeenahmed/magictap && cd magictap
uv sync
uv run magictap setup
uv run magictap recenter # at the start of each session
uv run magictap buttons- A Mac laptop with Apple Silicon and its built-in camera and microphone, macOS 14 or later (developed on an M4 Max). Optimized for laptops: it relies on the camera sitting just above the screen.
- Python 3.12 (
uvinstalls it for you) - Permissions for your terminal app, in System Settings > Privacy & Security:
- Camera and Microphone: always needed
- Accessibility: only for
magictap mouse, which clicks for you
magictap doctor checks all of this and tells you what to fix.
| Command | What it does |
|---|---|
magictap setup |
Guided first run: doctor, then calibrate, taps and recenter |
magictap recenter |
Start of every session (15 s): look at and touch 6 dots; corrects drift from seating, lid angle and light |
magictap doctor |
Checks camera, mic, permissions, models and calibration |
magictap calibrate |
Eye calibration, plus touch rounds on a grid. The eye mapping is trained on all your past calibrations together |
magictap taps |
Teaches it your screen taps vs. typing, talking and noise |
magictap buttons |
Big on-screen buttons with right/wrong feedback. Guided mode keeps learning as you use it |
magictap mouse |
Use it in any app: tap the screen and it clicks where you're looking |
magictap serve |
Local WebSocket service for your app (see below) |
magictap preview |
Camera view with the face and eye overlay, for checking framing |
magictap evaluate-demo |
Offline accuracy on a recording from magictap record-demo (with and without re-centre) |
magictap mouseLook at something and tap the screen on it, and Magic Tap clicks there. This works with any app whose buttons are big enough (see Results for what "big enough" means).
magictap serve # ws://127.0.0.1:8765Your app tells Magic Tap where its buttons are and gets an event when one is tapped. Coordinates are global screen points, with the origin at the top left of the main display.
| Direction | Message | Meaning |
|---|---|---|
| app → Magic Tap | {"type":"targets","targets":[{"id":"next","x":1200,"y":300,"w":400,"h":300}]} |
Your buttons (replaces the previous set) |
| app → Magic Tap | {"type":"subscribe","gaze":true} |
Also stream the gaze point (~15 Hz) |
| Magic Tap → app | {"type":"hello","version":"0.2.0","screen":[1728,1117]} |
Sent on connect |
| Magic Tap → app | {"type":"tap","x":1395,"y":452,"target":"next","confidence":0.92,"t":…} |
A tap: where you were looking, which of your buttons that is (null if none) and how sure it is (0–1) |
| Magic Tap → app | {"type":"ignored","reason":"not a tap"} |
A sound that was rejected: not a tap, unusual sound, keyboard/trackpad or eyes not on screen |
| Magic Tap → app | {"type":"gaze","x":…,"y":…} |
Only if subscribed |
JavaScript (clients/js/magictap.js, no dependencies):
import { connect, screenRect } from "./magictap.js";
const mt = connect(); // reconnects automatically
mt.setTargets([{ id: "next", ...screenRect(document.querySelector("#next")) }]);
mt.on("tap", (tap) => { if (tap.target === "next" && tap.confidence > 0.6) nextSlide(); });Python (magictap.client):
from magictap.client import MagicTapClient
client = MagicTapClient()
client.set_targets([{"id": "next", "x": 1200, "y": 300, "w": 400, "h": 300}])
client.on_tap = lambda tap: print(tap["target"], tap["confidence"])
client.run_forever()Examples
examples/web/: a full-screen page with six buttons. Runmagictap serve, thenpython3 -m http.server -d examples/web 8000and open http://localhost:8000. Pages need to be served over http, because browsers block module imports fromfile://.examples/python_client.py: reports left or right half taps.
Any language with WebSockets works the same way, including Swift's
URLSessionWebSocketTask.
- Where: your eyes.
- MediaPipe face mesh, with the face normalised for head pose.
- ETH-XGaze gaze estimation (ResNet-18).
- A per-user mapping to screen points, traine 74F1 d on all your calibration sessions together, so it isn't tied to one day's posture or light.
- A 15-second re-centre per session fits an affine correction for today's seating, lid angle and lighting.
- While you touch the screen your gaze reads slightly high; Magic Tap learns that shift and subtracts it.
- When: the tap sound.
- 3 ms onset detection against an adaptive noise floor, with clips aligned to the exact sample of impact.
- A classifier trained on your taps vs. your everyday noises.
- An "unusual sound" check.
- Sounds within 150 ms of a key or trackpad press are ignored.
- Fusion: a tap is placed where your eyes last settled (a fixation) in the 0.8 s before it, which skips the glance from the previous button. If fewer than 60% of those gaze samples agree on a button, Magic Tap says "not sure" instead of clicking.
CONFIGURATION.md has the exact settings and what we learned building it.
| Right button on a 3×2 grid, new session, after a 6-dot re-centre | 18 / 18 (100 %) on held-out presses |
| Same, without the re-centre | 11 / 24 (46 %) |
| Gaze error on a session it never saw (3 sessions, leave-one-out, after re-centre) | 148 pt median |
| Taps detected | 91–95 % |
| False taps from typing, talking, knocks and music | 0 of 73 |
How the 18/18 was measured: an offline evaluation of a recorded session from a
different day than any calibration. The first press of each of the 6 cells
stood in for magictap recenter, and the other 18 presses were scored with
nothing from them used in fitting. Reproduce it with magictap evaluate-demo.
Caveats: it's a small sample from one person (the 95 % confidence interval for 18/18
starts at about 81 %), it's offline rather than live, and the presses were finger
rests rather than heard taps. Live validation with magictap buttons after a
re-centre is next. Magic Tap is built for a handful of large buttons, not precise
pointing.
- Nothing happens / it never says "ready": run
magictap doctor. Camera and Microphone must be allowed for the app you run it from (Terminal, iTerm, VS Code…). - Wrong button: run
magictap recenter(15 s); that fixes most of it. If it keeps happening, recalibrate in the position you actually use. Keep text and camera previews out of the middle of the screen while calibrating; anything readable pulls your eyes away from the dots. - Taps not heard, or heard when you didn't tap: run
magictap tapsagain in the room you'll use it in. Tap with your fingertip; gentle taps work, but firm ones are more reliable. - Clicks in mouse mode don't happen: allow Accessibility for your terminal app.
- Gaze model: ETH-XGaze ResNet-18 weights by hysts
(ptgaze, MIT). The 3D face template
in
magictap/assetscomes from the same project; its licence is included. - Face, pose and hand landmarks: MediaPipe.
- Tap-detector ideas (short-window onsets, exact alignment, novelty rejection and learning from evaluation taps): JustinGamer191/Holo and its forks.
MIT, see LICENSE.