WebXR Hit Test Module

1. Introduction

This module describes a mechanism for allowing WebXR applications to cast rays into the users' real world environment and report back, to the best of the XR device’s knowledge, the point at which the ray intersected with a physical object along with the orientation of the intersected surface. This allows for virtual objects to be placed in alignment with those surfaces, such as placing objects realistically on the floor or attaching them to a wall. The hit test API is an extension to WebXR Device API and builds on top of WebXR Augmented Reality Module.

1.1. Terminology

Hit testing, as understood by this document, is an act of checking if an idealised mathematical ray (half-line) intersects with real world as understood by the underlying Augmented Reality hardware & software. Ray intersections against virtual objects created by the application consuming the API are explicitly out of scope of the hit test API.

2. Initialization

2.1. Feature descriptor

In order for the applications to signal their interest in performing hit testing during a session, the session must be requested with appropriate feature descriptor. The string hit-test is introduced by this module as a new valid feature descriptor for hit test feature.

A device is capable of supporting the hit test feature if the device exposes a native hit test capability. The inline XR device MUST NOT be treated as capable of supporting the hit test feature.

The hit test feature is subject to feature policy and requires "xr-spatial-tracking" policy to be allowed on the requesting document’s origin.

3. Hit test options

3.1. XRHitTestTrackableType

An XRHitTestTrackableType enum specifies the type of entity that can be used for the purposes of hit test source creation.

enum XRHitTestTrackableType {
  "point",
  "plane",
  "mesh"
};

A hit test trackable of type "point" indicates that the hit test results will be computed based on the feature points detected by the underlying Augmented Reality system.
A hit test trackable of type "plane" indicates that the hit test results will be computed based on the planes detected by the underlying Augmented Reality system.
A hit test trackable of type "mesh" indicates that the hit test results will be computed based on the meshes detected by the underlying Augmented Reality system.

3.2. XRHitTestOptionsInit

An XRHitTestOptionsInit dictionary represents a set of configurable values that affect the behavior of the hit test being performed.

dictionary XRHitTestOptionsInit {
  required XRSpace space;
  sequence<XRHitTestTrackableType> entityTypes;
  XRRay offsetRay;
};

The space dictionary member specifies XRSpace relative to which the offsetRay is specified.

The entityTypes dictionary member specifies array of XRHitTestTrackableTypes that will be used to compute results of the hit test.

The offsetRay dictionary member specifies XRRay that will be used to perform hit test. The offsetRay will be interpreted as if expressed in a coordinate system defined by space.

The XRHitTestOptionsInit dictionary has an associated effective entityTypes which is set to entityTypes if it was provided at dictionary construction time. If the entityTypes was not provided at construction time, the effective entityTypes is set to an array containing single element, "plane".

The XRHitTestOptionsInit dictionary has an associated effective offsetRay which is set to offsetRay if it was provided at dictionary construction time. If the offsetRay was not provided at construction time, the effective offsetRay is set to an XRRay constructed by invoking XRRay() without any parameters.

3.3. XRTransientInputHitTestOptionsInit

An XRTransientInputHitTestOptionsInit dictionary represents a set of configurable values that affect the behavior of the hit test for transient input that is being performed.

dictionary XRTransientInputHitTestOptionsInit {
  required DOMString profile;
  sequence<XRHitTestTrackableType> entityTypes;
  XRRay offsetRay;
};

The profile dictionary member specifies an input profile name of the transient input source that will be used to compute hit test results.

The entityTypes dictionary member specifies array of XRHitTestTrackableTypes that will be used to compute results of the hit test.

The offsetRay dictionary member specifies XRRay that will be used to perform hit test. The offsetRay will be interpreted as if expressed in a coordinate system defined by XRInputSource whose profile matches the passed in profile when computing hit test results for transient input sources.

The XRTransientInputHitTestOptionsInit dictionary has an associated effective entityTypes which is set to entityTypes if it was provided at dictionary construction time. If the entityTypes was not provided at construction time, the effective entityTypes is set to an array containing single element, "plane".

The XRTransientInputHitTestOptionsInit dictionary has an associated effective offsetRay which is set to offsetRay if it was provided at dictionary construction time. If the offsetRay was not provided at construction time, the effective offsetRay is set to an XRRay constructed by invoking XRRay() without any parameters.

4. Hit test source

4.1. XRHitTestSource

[SecureContext, Exposed=Window]
interface XRHitTestSource {
  undefined cancel();
};

The XRHitTestSource object serves as a handle to an active subscription to hit test.

Each XRHitTestSource has an associated session which stores an XRSession that was used to create the hit test source.

Each XRHitTestSource has an associated native origin which stores information sufficient to identify the native origin of an XRSpace that was used to request hit test. This information will be subsequently used when computing hit test results.

Each XRHitTestSource has an associated entity types, which is an array of XRHitTestTrackableTypes describing which entity types will be considered when computing hit test results.

Each XRHitTestSource has an associated offset ray, which is an XRRay that will be used when computing hit test results.

XRHitTestSource is considered active for as long as it’s present in session's set of active hit test sources.

In order to create a hit test source from session, space, entityTypes and offsetRay, the user agent MUST run the following steps:

Let hitTestSource be a new XRHitTestSource.
Initialize hitTestSource’s session to session.
Initialize hitTestSource’s native origin to space’s native origin.
Initialize hitTestSource’s entity types to entityTypes.
Compute transformedOffsetRay from offsetRay and space such that transformedOffsetRay, when interpreted in space’s native origin coordinate system, represents the same ray as offsetRay does when interpreted in space’s effective origin coordinate system.
Initialize hitTestSource’s offset ray to transformedOffsetRay.
Return hitTestSource.

The cancel() method, when invoked on XRHitTestSource hitTestSource, signals that the application is no longer interested in obtaining hit test results for the specified hitTestSource.

When cancel() method is invoked, the user agent MUST cancel a hit test source by running the following steps:

If the hitTestSource is not active, throw an InvalidStateError and abort these steps.
Remove hitTestSource from session's set of active hit test sources.

When the application no longer retains any references to a particular XRHitTestSource hitTestSource, the user agent MAY cancel a hit test source if hitTestSource is still active. The cancelation MAY happen at an unspecified time (or not at all) and the application SHOULD NOT rely on this behavior for cleanup.

4.2. XRTransientInputHitTestSource

[SecureContext, Exposed=Window]
interface XRTransientInputHitTestSource {
  undefined cancel();
};

The XRTransientInputHitTestSource object serves as a handle to an active subscription to hit test for transient input sources.

Each XRTransientInputHitTestSource has an associated session which stores an XRSession that was used to create the hit test source.

Each XRTransientInputHitTestSource has an associated profile which stores input profile name of an input source. This information will be subsequently used when computing hit test results for transient input sources.

Each XRTransientInputHitTestSource has an associated entity types, which is an array of XRHitTestTrackableTypes describing which entity types will be considered when computing hit test results.

Each XRTransientInputHitTestSource has an associated offset ray, which is an XRRay that will be used when computing hit test results.

XRTransientInputHitTestSource is considered active for as long as it’s present in session's set of active hit test sources for transient input.

In order to create a hit test source for transient input from session, profile, entityTypes and offsetRay, the user agent MUST run the following steps:

Let hitTestSource be a new XRTransientInputHitTestSource.
Initialize hitTestSource’s session to session.
Initialize hitTestSource’s profile to profile.
Initialize hitTestSource’s entity types to entityTypes.
Initialize hitTestSource’s offset ray to offsetRay.
Return hitTestSource.

The cancel() method, when invoked on XRTransientInputHitTestSource hitTestSource, signals that the application is no longer interested in obtaining hit test results for the specified hitTestSource.

When cancel() method is invoked, the user agent MUST cancel a hit test source for transient input by running the following steps:

If the hitTestSource is not active, throw an InvalidStateError and abort these steps.
Remove hitTestSource from session's set of active hit test sources for transient input.

When the application no longer retains any references to a particular XRTransientInputHitTestSource hitTestSource, the user agent MAY cancel a hit test source for transient input if hitTestSource is still active. The cancelation MAY happen at an unspecified time (or not at all) and the application SHOULD NOT rely on this behavior for cleanup.

5. Hit test result

5.1. XRHitTestResult

[SecureContext, Exposed=Window]
interface XRHitTestResult {
  XRPose? getPose(XRSpace baseSpace);
};

A XRHitTestResult contains single result of a hit test. It encapsulates information about the intersection point of the ray used to perform the hit test with user’s environment as understood by the underlying XR device.

Each XRHitTestResult has an associated frame which is an XRFrame for which the result was computed.

Each XRHitTestResult has an associated native origin. This native origin defines new coordinate system whose Y axis represents the surface’s normal vector at the intersection point.

In order to create a hit test result given XRFrame frame, array of XRHitTestTrackableType entityTypes, and native hit test result nativeResult, the user agent MUST run the following steps:

Let hitTestResult be a new XRHitTestResult.
Let session be frame’s session.
Let device be session’s XR device.
Query device for native entity type, nativeEntityType, of the nativeResult.
Convert from native entity type nativeEntityType to entityType.
If entityType is null or is not present in entityTypes array, return null and abort these steps.
Set hitTestResult’s frame to frame.
Set hitTestResult’s native origin to a native origin obtained from nativeResult.
Return hitTestResult.

The getPose(baseSpace) method, when invoked on XRHitTestResult hitTestResult with baseSpace parameter, provides the pose of the hitTestResult relative to baseSpace as an XRPose, at the time represented by frame.

When getPose(baseSpace) method is invoked on hitTestResult, the user agent MUST run the following steps:

Let frame be the hitTestResult’s frame.
If frame’s active boolean is false, throw an InvalidStateError and abort these steps.
Let pose be a new XRPose.
Let space be a new XRSpace, with native origin set to native origin, origin offset set to identity transform, and session set to frame's session.
Populate the pose of space in baseSpace at the time represented by frame into pose.
Return pose.

5.2. XRTransientInputHitTestResult

[SecureContext, Exposed=Window]
interface XRTransientInputHitTestResult {
  [SameObject] readonly attribute XRInputSource inputSource;
  readonly attribute FrozenArray<XRHitTestResult> results;
};

A XRTransientInputHitTestResult contains array of result of a hit test for transient input, grouped by XRInputSource inputSource.

The inputSource attribute contains an XRInputSource that was used to compute the results array.

The results attribute contains an array of computed XRHitTestResults.

Each XRTransientInputHitTestResult has an associated frame which is an XRFrame for which the results were computed.

In order to create a hit test result for transient input given XRInputSource inputSource, XRFrame frame, array of XRHitTestTrackableType entityTypes, and array of native hit test results nativeResults, the user agent MUST run the following steps:

Let hitTestResult be a new XRTransientInputHitTestResult.
Set hitTestResult’s frame to frame.
Set hitTestResult’s inputSource to inputSource.
Let results be an empty array of XRHitTestResults.
For each nativeResult in nativeResults:
1. Create a hit test result result from frame, entityTypes, and nativeResult.
2. If result is null, continue to the next entry in nativeResults.
3. Add result to results array.
Set hitTestResult’s results to results.
Return hitTestResult.

6. Requesting hit test

partial interface XRSession {
  Promise<XRHitTestSource> requestHitTestSource(XRHitTestOptionsInit options);
  Promise<XRTransientInputHitTestSource> requestHitTestSourceForTransientInput(XRTransientInputHitTestOptionsInit options);
};

The XRSession is extended to contain an associated set of active hit test sources that will be used when computing hit test results.

The XRSession is extended to contain an associated set of active hit test sources for transient input that will be used when computing hit test results for transient input.

The application is considered to make an unreasonable number of requests when the total number of active hit test sources or recently made requests is considered too high for a legitimate use of the API. This is an OPTIONAL privacy measure that the user agent MAY take to avoid abuse.

The application can request hit test using XRSession's requestHitTestSource() method.

The requestHitTestSource(options) method, when invoked on an XRSession session, MUST run the following steps:

Let promise be a new Promise.
If hit-test feature descriptor is not contained in the session’s list of enabled features, reject promise with NotSupportedError and abort these steps.
If session’s ended value is true, throw an InvalidStateError and abort these steps.
The user agent MAY reject promise with NotAllowedError and abort these steps if there is a unreasonable number of requests.
Add compute all hit test results algorithm to session’s list of frame updates if it is not already present there.
Create a hit test source, hitTestSource, with session, options’ space, options’ effective entityTypes and options’ effective offsetRay.
If hitTestSource is null, reject promise with an OperationError and abort these steps.
Store created hitTestSource in session’s set of active hit test sources.
Resolve promise with created hitTestSource.

The requestHitTestSourceForTransientInput(options) method, when invoked on an XRSession session, MUST run the following steps:

Let promise be a new Promise.
If hit-test feature descriptor is not contained in the session’s list of enabled features, reject promise with NotSupportedError and abort these steps.
If session’s ended value is true, throw an InvalidStateError and abort these steps.
The user agent MAY reject promise with NotAllowedError and abort these steps if there is a unreasonable number of requests.
Add compute all hit test results algorithm to session’s list of frame updates if it is not already present there.
Create a hit test source for transient input, hitTestSource, with session, options’ profile, options’ effective entityTypes and options’ effective offsetRay.
If hitTestSource is null, reject promise with an OperationError and abort these steps.
Store created hitTestSource in session’s set of active hit test sources for transient input.
Resolve promise with created hitTestSource.

7. Computing hit test results

In order to compute all hit test results for a given XRFrame frame, the user agent MUST perform the following steps:

Invoke compute hit test results algorithm with frame.
Invoke compute hit test results for transient input algorithm with frame.

In order to compute hit test results for a given XRFrame frame, for each hit test source, hitTestSource, that is present in session's set of active hit test sources, the user agent MUST perform the following steps:

Let entityTypes be the hitTestSource’s entity types.
Let session be frame’s session.
Let device be the session’s XR device.
Query the device’s tracking system for hitTestSource’s native origin's latest coordinates.
Interpret hitTestSource’s offset ray, offsetRay, as if expressed relative to coordinates and using that interpretation, perform native hit test obtaining native hit test results nativeResults.
Let hitTestResults be an empty list.
For each native hit test result nativeResult in nativeResults, perform the following steps:
1. Create a hit test result, hitTestResult from frame, entityTypes, and nativeResult.
2. If hitTestResult is null, continue to the next entry in nativeResults.
3. Add hitTestResult to hitTestResults such that the list remains sorted by the distance along the ray from offsetRay to nativeResult.
Store hitTestResults in frame’s map of hit test sources to hit test results under the hitTestSource key.

In order to compute hit test results for transient input for a given XRFrame frame, for each hit test source, hitTestSource, that is present in session's set of active hit test sources for transient input, the user agent MUST perform the following steps:

Let entityTypes be the hitTestSource’s entity types.
Let session be frame’s session.
Let device be the session’s XR device.
Let candidateInputSources be a set of all session’s input sources contained in list of active XR input sources that are considered transient.
Let matchingInputSources be a set of all input sources contained in candidateInputSources whose profiles array contain an entry equal to hitTestSource’s profile.
Let hitTestResults be an empty array of XRTransientInputHitTestResults.
For each transient input source inputSource in matchingInputSources:
1. Query the device’s tracking system for inputSource’s targetRaySpace's native origin's latest coordinates.
2. Interpret hitTestSource’s offset ray, as if expressed relative to coordinates and using that interpretation, perform native hit test obtaining native hit test results nativeResults.
3. Create a hit test result for transient input, hitTestResult from frame, inputSource, entityTypes, and nativeResults.
4. Add hitTestResult to hitTestResults array.
Store hitTestResults in frame’s map of hit test sources to hit test results for transient input under the hitTestSource key.

8. Obtaining hit test results

partial interface XRFrame {
  sequence<XRHitTestResult> getHitTestResults(XRHitTestSource hitTestSource);
  sequence<XRTransientInputHitTestResult> getHitTestResultsForTransientInput(XRTransientInputHitTestSource hitTestSource);
};

The XRFrame is extended to contain an associated map of hit test sources to hit test results that stores a mapping from XRHitTestSource to an array of XRHitTestResults.

The XRFrame is extended to contain an associated map of hit test sources to hit test results for transient input that stores a mapping from XRTransientInputHitTestSource to an array of XRTransientInputHitTestResults.

The application can obtain hit test results from an XRHitTestSource by using XRFrame's getHitTestResults() method.

When the getHitTestResults(hitTestSource) method, when invoked on an XRFrame frame, the user agent MUST run the following steps:

If frame’s active boolean is false, throw an InvalidStateError and abort these steps.
Check that the entry for hitTestSource is present in frame’s map of hit test sources to hit test results. If the entry is not present, throw an InvalidStateError and abort these steps.
Look up an entry for hitTestSource in frame’s map of hit test sources to hit test results and assign it to results.
Return results.

The application can obtain hit test results for transient input from an XRTransientInputHitTestSource by using XRFrame's getHitTestResultsForTransientInput() method.

When the getHitTestResultsForTransientInput(hitTestSource) method, when invoked on an XRFrame frame, the user agent MUST run the following steps:

If frame’s active boolean is false, throw an InvalidStateError and abort these steps.
Check that the entry for hitTestSource is present in frame’s map of hit test sources to hit test results for transient input. If the entry is not present, throw an InvalidStateError and abort these steps.
Look up an entry for hitTestSource in frame’s map of hit test sources to hit test results for transient input and assign it to results.
Return results.

9. Geometric primitives

9.1. XRRayDirectionInit

An XRRayDirectionInit dictionary represents a direction vector to be passed to the XRRay(origin, direction) constructor.

dictionary XRRayDirectionInit {
  double x = 0;
  double y = 0;
  double z = -1;
  double w = 0;
};

9.2. XRRay

An XRRay is a geometric ray described by an origin point and direction vector.

An XRRay contains a matrix which is a matrix.

[SecureContext, Exposed=Window]
interface XRRay {
  constructor(optional DOMPointInit origin = {}, optional XRRayDirectionInit direction = {});
  constructor(XRRigidTransform transform);
  [SameObject] readonly attribute DOMPointReadOnly origin;
  [SameObject] readonly attribute DOMPointReadOnly direction;
  [SameObject] readonly attribute Float32Array matrix;
};

The XRRay(origin, direction) constructor MUST perform the following steps when invoked:

Let ray be a new XRRay.
Initialize ray’s origin to { x: 0.0, y: 0.0, z: 0.0, w: 1.0 }.
Initialize ray’s direction to { x: 0.0, y: 0.0, z: -1.0, w: 0.0 }.
If all of direction’s x, y, and z are zero, throw a TypeError and abort these steps.
If direction’s w is not 0.0, throw a TypeError and abort these steps.
If origin’s w is not 1.0, throw a TypeError and abort these steps.
Initialize ray’s origin’s x value to origin’s x, y value to origin’s y, and z value to origin’s z.
Initialize ray’s direction’s x value to direction’s x, y value to direction’s y, and z value to direction’s z.
Normalize the x, y, and z components of ray’s direction.
Initialize ray’s matrix to null.
Return ray.

The XRRay(transform) constructor MUST perform the following steps when invoked:

Let ray be a new XRRay.
Initialize ray’s origin to { x: 0.0, y: 0.0, z: 0.0, w: 1.0 }.
Initialize ray’s direction to { x: 0.0, y: 0.0, z: -1.0, w: 0.0 }.
Transform ray’s origin by premultiplying the transform’s matrix and set ray to the result.
Transform ray’s direction by premultiplying the transform’s matrix and set ray to the result.
Normalize the x, y, and z components of ray’s direction.
Initialize ray’s matrix to null.
Return ray.

The origin attribute defines the 3-dimensional point in space that the ray originates from, given in meters. The origin's w attribute MUST be 1.0.

The direction attribute defines the ray’s 3-dimensional directional vector. The direction's w attribute MUST be 0.0 and the vector MUST be normalized to have a length of 1.0.

The matrix attribute is a matrix which represents a transform that can be used to position objects along the XRRay. It is a transform from a ray originating at [0, 0, 0] and extending down the negative Z axis to the ray described by the XRRay's origin and direction. Such a matrix MUST be one that has a rotation component which leaves any vector perpendicular to direction and the Z axis unchanged. This attribute MUST be computed by obtaining the matrix for the XRRay. This attribute SHOULD be lazily evaluated.

Note: The XRRay's matrix can be used to easily position graphical representations of the ray when rendering.

To obtain the matrix for a given XRRay ray

If ray’s matrix is not null, perform the following steps:
1. If the operation IsDetachedBuffer on matrix is false, return ray’s matrix.
Let z be the vector [0, 0, -1].
Let axis be the vector cross product of z and ray’s direction, z × direction.
Let cos_angle be the scalar dot product of z and ray’s direction, z · direction.
Set rotation based on the following:

If cos_angle is greater than -1 and less than 1
Set rotation to the rotation matrix representing a right handed planar rotation around axis by arccos(cos_angle).
Else, if cos_angle is -1
Set rotation to the rotation matrix representing a right handed planar rotation around vector [1, 0, 0] by arccos(cos_angle).
Else
Set rotation to an identity matrix.
Let translation be the translation matrix with components corresponding to ray’s origin.
Let matrix be the result of premultiplying rotation from the left onto translation (i.e. translation * rotation) in column-vector notation.
Set ray’s matrix to matrix.
Return matrix.

The distance along the ray, distance, from XRRay ray to any entity entity is defined such that ray.origin + ray.direction * distance results in a point beloning to the entity entity, distance is non-negative, and there does not exist a smaller value of distance for the above predicate to still hold. It is up to the XR device to define the meaning of "point belonging to an entity".

10. Native device concepts

User agents implementing hit test API must have a way of obtaining information about user’s environment from underlying XR device. This section attempts to describe requirements and concepts related to native capabilities of the device and is by neccesity sufficiently under-specified to leave ample room for different underlying frameworks / devices.

10.1. Native hit test

In this specification it is assumed that XR device exposes a way for the user agent to perform a native hit test that satisfies the following requirements:

Accepts a 3D ray that will be tested against user’s environment.
Returns a collection of 3D poses representing intersection points of the passed in ray with user’s environment. Each entry in the collection should also contain information about the type of the native entity that was used to obtain that native result and enough information to enable the user agent to compute surface normal to the user’s environment at the intersection point.

Note: For devices that do not expose the hit test functionality natively, it might still be possible for user agents to implement this specification by leveraging other ways of obtaining the information about user’s environment that might be exposed by the XR device.

10.2. Native entity type

Native hit test results returned by XR device should contain information about the type of the entity used to compute the result. Such native types might consist of, but not be limited to:

Point - signifies that hit test result was computed based on characteristic points found in user’s environment.
Plane - signifies that hit test result was computed based on planes detected in user’s environment.
Mesh - signifies that the hit test result was computed based on meshes detected in user’s environment.

To convert from native entity type into XRHitTestTrackableType, the user agent MUST run the following steps:

Let nativeEntityType be the native entity type to be converted.
Let entityType be a new XRHitTestTrackableType.
Initialize entityType as follows:

If nativeEntityType contains type that corresponds to "point"
Set entityType to "point".
Else, if nativeEntityType contains type that corresponds to "plane"
Set entityType to "plane".
Else, if nativeEntityType contains type that corresponds to "mesh"
Set entityType to "mesh".
Else
Set entityType to null
Return entityType.

10.3. Native hit test result

Native hit test results returned from XR device should contain the position of the intersection point with user’s environment. Depending on the native entity type and the information available to the XR device, the result should also contain orientation defined in such a way to allow the user agent to compute a surface normal to the user’s environment at the intersection point.

The information about position and orientation of the intersection point should be contained in native hit test result's native origin. Native origin defines a new coordinate system in such a way that its Y axis represents the surface’s normal vector at the intersection point. If the orientation is not returned from the XR device, the user agent SHOULD set the native origin in such a way that Y axis of the coordinate system it defines is pointing up (towards negative gravity vector).

Decide if we need to specify other axes of the coordinate system defined by hit test result’s native origin to maintain compatibility between different implementations & differrent AR frameworks.

11. Privacy & Security Considerations

The hit test API can be used to map the user environment with more or less precision by sending hit test rays in multiple directions. Hit test has to be declared when creating an XR session as a feature descriptor which will allow the user agent to notify the user of the potential privacy implications of allowing the hit test API to be used by the website. Furthermore, the user agent is allowed to deny hit test requests when it considers that an unreasonable number of requests have been made for a genuine non-privacy invasive usage.

Changes

Changes from the First Public Working Draft 31 August 2021

12. Acknowledgements

The following individuals have contributed to the design of the WebXR Hit Test specification: