WO2025098400A1 - Camera parameter calibration method and related device - Google Patents

Abstract

Provided in the present disclosure are a camera parameter calibration method and a related device. The method comprises: receiving a plurality of images obtained by means of at least two cameras photographing a calibration reference object, wherein the at least two cameras are provided in a lens housing of a wearable device; determining a projection relationship between pixel points of the plurality of images and the calibration reference object; and determining a first parameter set and a second parameter set of the at least two cameras on the basis of the projection relationship, wherein the first parameter set comprises a plurality of first parameters, the first parameters are used for representing a target pixel point in the images captured by a camera to be subjected to calibration and a projection direction corresponding to the target pixel point, the second parameter set comprises at least one group of second parameters, and the at least one group of second parameters is used for indicating a pose relationship between the at least two cameras.

Description

Camera parameter calibration method and related equipment

This application claims priority to the Chinese invention patent application entitled “Camera parameter calibration method and related equipment” filed on November 8, 2023 and application number 202311484874.4, the entire contents of which are incorporated by reference into this application.

Technical Field

The present disclosure relates to the field of extended reality technology, and in particular to a camera parameter calibration method and related equipment.

Background Art

Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR (extended reality) technology can further include augmented reality (AR), virtual reality (VR), and mixed reality (MR), using hardware devices combined with a variety of technical means to integrate virtual content and real scenes.

Generally, the extended reality system provides a wearable device for the user to realize human-computer interaction, and the wearable device may be a head-mounted wearable device. In some scenarios, the wearable device may realize the line of sight tracking or pupil distance estimation function by collecting human eye images for calculation.

However, in the related art, the camera for collecting images is usually set outside the lens barrel, which limits the accuracy of pupil distance estimation and gaze tracking algorithms.

Summary of the invention

The present disclosure proposes a camera parameter calibration method and related devices to solve or partially solve the above problems.

In a first aspect, the present disclosure provides a method for calibrating camera parameters, comprising:

Receiving a plurality of images obtained by photographing a calibration reference object by at least two cameras, wherein the at least two cameras are arranged in a lens barrel of a wearable device; the wearable device further comprises a binocular display module, the lens barrel is arranged on a light-emitting side of the binocular display module, an optical component is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical component;

Determining a projection relationship between pixel points of the plurality of images and the calibration reference object;

Determining a first parameter set and a second parameter set of the at least two cameras according to the projection relationship;

Among them, the first parameter set includes multiple first parameters, the first parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one group of second parameters, and the at least one group of second parameters is used to indicate the posture relationship between the at least two cameras.

In a second aspect of the present disclosure, a wearable device is provided, comprising:

Binocular display module;

Two lens barrels are arranged on the light-emitting side of the binocular display module, and at least one of the two lens barrels includes at least two cameras and an optical component arranged inside the lens barrel for collecting human eye images, and the at least two cameras are located between the binocular display module and the optical component.

In a third aspect of the present disclosure, a camera parameter calibration device is provided, comprising:

The receiving module is configured to: receive a plurality of images obtained by photographing a calibration reference object by at least two cameras; the wearable device further comprises a binocular display module, the lens barrel is arranged on a light-emitting side of the binocular display module, an optical component is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical component;

A first determination module is configured to: determine a projection relationship between pixel points of the plurality of images and the calibration reference object;

A second determination module is configured to: determine a first parameter set and a second parameter set of the at least two cameras according to the projection relationship;

Among them, the first parameter set includes multiple first parameters, the first parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one group of second parameters, and the at least one group of second parameters is used to indicate the posture relationship between the at least two cameras.

In a fourth aspect of the present disclosure, a wearable device is provided, comprising:

A lens barrel, wherein a display module, an optical component, and at least two cameras are arranged in the lens barrel, wherein the at least two cameras and the optical component are located at a light-emitting side of the display module, and the at least two cameras are located between the optical component and the display module;

A processing module is electrically coupled to the at least two cameras and is configured to: obtain a first parameter set and a second parameter set obtained by the method of the first aspect and an image taken by the camera; and use the first parameter set, the second parameter set and the image to solve the position of the target area in the image in space.

In a fifth aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect.

In a sixth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

In a seventh aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG. 1A shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

FIG. 1B shows a schematic diagram of an exemplary head-mounted wearable device.

1C and 1D are schematic diagrams showing exemplary human eye images.

FIG. 2A shows a schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure.

FIG. 2B shows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

FIG. 2C shows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

FIG. 2D shows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

FIG. 3A shows a schematic flow chart of an exemplary method provided in an embodiment of the present disclosure.

FIG. 3B shows a flowchart of an exemplary method for determining a projection relationship according to an embodiment of the present disclosure.

FIG4A shows a schematic diagram of an exemplary image collection scenario according to an embodiment of the present disclosure.

FIG4B shows a schematic diagram of another exemplary image collection scenario according to an embodiment of the present disclosure.

FIG. 4C is a schematic diagram showing three selected target images.

FIG4D is a schematic diagram showing a projection relationship between a pixel point and a calibration reference object according to an embodiment of the present disclosure.

FIG. 4E shows a schematic diagram of an exemplary first parameter according to an embodiment of the present disclosure.

FIG. 4F shows a schematic diagram of a calibration principle of a second parameter of an exemplary camera in a different-side lens barrel according to an embodiment of the present disclosure.

FIG. 4G is a schematic diagram showing an image obtained after the captured image is binarized in an embodiment of the present disclosure.

FIG5 shows a schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure.

FIG. 6 shows a schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of the present disclosure.

FIG. 7 shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

FIG. 1A shows a schematic diagram of an exemplary extended reality system 100 provided by an embodiment of the present disclosure.

Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR (extended reality) technology can further include augmented reality (AR), virtual reality (VR), and mixed reality (MR), using hardware devices combined with a variety of technical means to integrate virtual content and real scenes.

As shown in FIG1A , the system 100 may include various types of wearable devices, such as a head-mounted wearable device (e.g., VR/AR glasses or a head-mounted display (HMD)) 104, an operating handle 108, etc. In some scenarios, a camera/camera 110 for taking photos of the operator (user) 102 may also be provided. In some embodiments, when the aforementioned device does not have a processing function, the system 100 may also include an external control device 112 for providing a processing function. The control device 112, for example, may be a computer device such as a mobile phone or a computer. In some embodiments, when any of the aforementioned devices is used as a control device or a main control device, it may communicate with other devices in the system 100 through wired or wireless communication methods to achieve information interaction.

In the system 100, the user 102 can use the head-mounted wearable device 104 and the operating handle 108 to interact with the extended reality system 100. In some scenarios, the system 100 can use the images captured by the camera/camera 110 to recognize the posture and gesture of the user 102, and then complete the interaction with the user 102 based on the recognized posture and gesture. In some embodiments, the user 130 can also implement gesture input through bare hands, and the head-mounted wearable device 104 can collect the front image in real time through the camera or camera set in front of the head-mounted wearable device 104, and recognize the gesture of the user 130 by recognizing the image.

In some embodiments, as shown in FIG1A , the system 100 may also communicate with a server 114 and may obtain data, such as pictures, audio, video, etc., from the server 114 and may output these data through the head-mounted wearable device 104, such as displaying pictures or videos on the display screen of the head-mounted wearable device 104, playing audio and video carried by audio using the speakers of the head-mounted wearable device 104, etc. In some embodiments, as shown in FIG1A , the server 114 may retrieve required data, such as pictures, audio, video, etc., from a database server 116 for storing data.

In some embodiments, a collection unit for collecting information may be provided on the head-mounted wearable device 104. The types of the collection unit may be various.

In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning tracking unit, wherein the environment acquisition unit may be used to acquire environment information around (for example, in front of) the wearable device 104, and the positioning tracking unit may be used to position and track the wearable device 104. Optionally, the environment acquisition unit may include but is not limited to a three-color camera (for example, an RGB camera), a depth camera, a binocular camera, a laser and other photosensitive elements, and the positioning tracking unit may include but is not limited to visual simultaneous positioning and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), an ultra-wideband wireless communication technology (UWB), a laser and other modules.

In some embodiments, the head-mounted wearable device 104 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc., for collecting speed information or acceleration information of the head-mounted wearable device 104. For another example, the operating handle 108 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc., for collecting speed information or acceleration information of the operating handle 108. It should be noted that, in addition to being provided on the head-mounted wearable device 104 and the operating handle 108, the aforementioned collection unit may also be provided on a body part of the interactive user 102 directly by attachment without relying on a hardware device, so as to collect relevant information of the body part, such as speed, acceleration, or angular velocity information, or information collected by other sensors or collection units.

In some embodiments, the head-mounted wearable device 104 may also be provided with a camera or a camera for taking photos of the operator (user) 102 (eg, photos of the hands or feet) and images of the environment.

In some embodiments, the system 100 can identify the posture and gestures of the user 102 through the collected information, and then can perform corresponding interactions based on the identified user posture and gestures.

FIG. 1B shows a schematic diagram of an exemplary head-mounted wearable device 104 .

As shown in FIG. 1B , the head-mounted wearable device 104 may include a lens barrel 1042, and a display screen 1044 for displaying an image and an optical component 1046 for processing an optical path may be provided inside the lens barrel 1042. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B), and the combination of the plurality of lenses may project light emitted by the display screen 1044 into the human eye 1022, so that the human eye 1022 may see the image displayed on the display screen 1044. It is understood that FIG. 1B only exemplarily shows a single-sided structure of the head-mounted wearable device 104. In order to realize binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged in parallel.

In some embodiments, as shown in FIG. 1B , the head-mounted wearable device 104 may also be provided with a camera 1048 for capturing images of the human eye, and the camera 1048 may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

Optionally, the camera 1048 can be an eye tracking (ET) camera, and the human eye images collected by the camera can be used to realize functions such as pupil distance estimation and eye tracking.

As shown in FIG1B , in the related art, the camera 1048 is usually arranged outside the lens barrel and usually only one camera is arranged corresponding to each lens barrel. In addition, in order to better capture the complete human eye image and not affect the human eye's observation of the screen 1044, the common deployment position of the camera 1048 is generally at the outer corner of the eye or the nose wing. Referring to FIG1B , if the camera 1048 is close to the outside of the device, the camera deployment position shown in FIG1B is the outer corner of the eye, and if the camera 1048 is close to the inside of the device, the camera deployment position shown in FIG1B is the nose wing.

However, the camera installation method in the related art easily causes the camera 1048 to be installed at a large inclination angle relative to the human eye 1022, resulting in a large angle α between the direction of the camera 1048 and the normal viewing direction of the human eye 1022, making it difficult for the collected human eye image to reflect the image at the normal viewing angle of the human eye, as shown in Figures 1C and 1D.

In some cases, the user may need to wear glasses before using the head-mounted wearable device 104. However, since the camera 1048 is arranged outside the lens barrel 1042, the camera 1048 is higher than the lens barrel 1042, which is easy to squeeze the glasses, affecting the wearing comfort of the head-mounted wearable device 104. At the same time, the imaging of the camera 1048 is easily affected by the edge of the glasses. The light refracts through the edge of the glasses, which reduces the image clarity of the camera 1048 and forms many refracted light spots in the image, thereby affecting the accuracy of the subsequent algorithm. In particular, when there is only one camera corresponding to the lens barrel, such a problem will be further aggravated.

In view of this, an embodiment of the present disclosure provides a wearable device, in which at least two cameras are arranged inside the lens barrel, which can solve or partially solve the above-mentioned problems to a certain extent.

FIG. 2A shows a schematic diagram of an exemplary wearable device 200 provided in an embodiment of the present disclosure.

As shown in FIG2A , similarly, the wearable device 200 may also include a lens barrel 202, and a display module 204 and an optical component 206 disposed inside the lens barrel 202. The optical component 206 may further include a plurality of lenses (e.g., lenses 2062 and 2064), and the combination of the plurality of lenses may project the light emitted by the display module 204 into the human eye 1022, so that the human eye 1022 may view the image displayed by the display module 204.

Unlike the wearable device 104 shown in FIG. 1B , the wearable device 200 includes two cameras 208A and 208B, and both cameras 208A and 208B are arranged inside the lens barrel 202. Since the cameras 208A and 208B are placed inside the lens barrel 202, the wearing of the glasses will not be affected, thereby improving the comfort of the wearable device 200. At the same time, as shown in FIG. 2A , since the cameras 208A and 208B are arranged inside the lens barrel 202, the distance between the cameras 208A and 208B and the human eye 1022 is extended, so that the installation inclination angle of the cameras 208A and 208B relative to the human eye 1022 becomes smaller, and then the angle β between the orientation of the cameras 208A and 208B and the normal viewing direction of the human eye 1022 is smaller than the angle α, so that the cameras 208A and 208B have a better observation angle, and the collected human eye image can better reflect the image of the normal viewing angle of the human eye, and the imaging quality is better. Furthermore, since the cameras 208A and 208B are placed inside the lens barrel 202, the glasses will not interfere with the imaging of the cameras 208A and 208B, further improving the imaging quality. The improved imaging quality also improves the accuracy of algorithms such as pupil distance estimation or line of sight tracking. In addition, since two cameras 208A and 208B are used, more observation information can be provided (more images can be collected), which can also improve the accuracy of algorithms such as pupil distance estimation or line of sight tracking.

In some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or line of sight tracking, as shown in FIG2A , two cameras 208A and 208B may be symmetrically arranged in the lens barrel 202 relative to the axis (the central dotted line of FIG2A ) of the lens barrel 202. In this way, the images captured by the two cameras 208A and 208B may be symmetrical, and the processing efficiency of subsequent algorithms may be further improved.

In some embodiments, as shown in FIG2A , both cameras 208A and 208B face the light-emitting side of the lens barrel 202 and the angles between the camera 208A and 208B and the axis of the lens barrel 202 (the central dotted line of FIG2A ) are equal, for example, the angles are both β. In this way, the images collected by the two cameras 208A and 208B can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

FIG. 2B shows a schematic diagram of another exemplary wearable device 200 provided by an embodiment of the present disclosure.

As shown in FIG. 2B , in some embodiments, the wearable device 200 may further include two reflective structures 210A and 210B, which may be structures having a reflective surface such as a reflective film or a reflective mirror. The reflective structures 210A and 210B may correspond to cameras 208A and 208B, respectively. Both cameras 208A and 208B face the display module 204, and the reflective structure 210A is used to reflect the light from the human eye 1022 into the camera 208A, and the reflective structure 210B is used to reflect the light from the human eye 1022 into the camera 208B. Through the reflection of the light by the reflective structures 210A and 210B, the cameras 208A and 208B can still collect images of the human eye. Moreover, because a reflection process is added to the optical path, the observation angle γ is further reduced, and the cameras 208A and 208B can better perform imaging.

In some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or line of sight tracking, as shown in FIG2B , two cameras 208A and 208B may be symmetrically arranged in the lens barrel 202 relative to the axis (the central dotted line of FIG2B ) of the lens barrel 202, and the reflective structures 210A and 210B may also be symmetrically arranged in the lens barrel 202 relative to the axis (the central dotted line of FIG2B ) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B may be symmetrical, and the processing efficiency of the subsequent algorithms may be further improved.

In some embodiments, as shown in FIG2B , both cameras 208A and 208B face the display module 204 and the angles between the two cameras 208A and 208B and the axis of the lens barrel 202 (the central dotted line of FIG2B ) are equal, for example, the angles are both γ; at the same time, the angles between the reflection structures 210A and 210B and the axis of the lens barrel 202 (the central dotted line of FIG2B ) are also equal. In this way, the images collected by the two cameras 208A and 208B can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

The aforementioned embodiments are all described by arranging two cameras in a single lens barrel. It can be understood that when the number of cameras is further increased, the observation data can be further increased, thereby further improving the algorithm accuracy. Therefore, embodiments in which two or more cameras are arranged in a single lens barrel should all fall within the protection scope of the present disclosure.

FIG. 2A and FIG. 2B only exemplarily show a single-side structure of the wearable device 200 . It can be understood that in order to achieve binocular display, the wearable device 200 may include two lens barrel structures arranged in parallel.

FIG. 2C shows a schematic diagram of another exemplary wearable device 200 provided by an embodiment of the present disclosure.

As shown in Fig. 2C, the wearable device 200 may include a binocular display module, which may further include a first display module 204A and a second display module 204B. The first display module 204A and the second display module 204B may respectively display an image for observation by the first eye 1022A (e.g., right eye) and an image for observation by the second eye 1022B (e.g., left eye).

In some embodiments, as shown in FIG2C , the wearable device 200 may further include two lens barrels disposed on the light-emitting side of the binocular display module, for example, a first lens barrel 202A and a second lens barrel 202B. At least one of the two lens barrels may further include at least two cameras disposed inside the lens barrel for collecting human eye images, thereby providing more observation data and improving the accuracy of subsequent algorithms.

As an optional embodiment, as shown in FIG2C , the first lens barrel 202A can be arranged on the light-emitting side of the first display module 204A, and the first lens barrel 202A can further be provided with a first camera 208A and a second camera 208B for collecting the human eye image of the first eye 1022A. In this way, for the human eye image collected by the first eye 1022A (for example, the right eye), a more accurate calculation result can be obtained when it is subsequently used for pupil distance calculation or line of sight tracking. In some embodiments, as shown in FIG2C , the first optical assembly 206A can also be arranged in the first lens barrel 202A, for projecting the image displayed by the first display module 204A into the first eye 1022A through optical principles. Optionally, the first optical assembly 206A can further include a first lens 2062A and a second lens 2064A, and the parameters of the first lens 2062A and the second lens 2064A can be different or the same, and can be designed according to actual needs. It can be understood that the type and number of lenses in the first optical component 206A are variable, and the specific type and number of lenses used can be designed according to actual needs.

Similarly, as another optional embodiment, as shown in FIG2C, the second lens barrel 202B can be arranged on the light-emitting side of the second display module 204A, and the third camera 208C and the fourth camera 208D for collecting the human eye image of the second eye 1022B can be further arranged in the second lens barrel 202A. In this way, for the human eye image collected by the second eye 1022B (for example, the left eye), a more accurate calculation result can be obtained when it is used for pupil distance calculation or line of sight tracking in the future. Similarly, in some embodiments, as shown in FIG2C, the second optical component 206B can also be arranged in the second lens barrel 202B, for projecting the image displayed by the second display module 204B into the second eye 1022B through optical principles. Optionally, the second optical component 206B can further include a third lens 2062B and a fourth lens 2064B, and the parameters of the third lens 2062B and the fourth lens 2064B can be different or the same, and can be designed according to actual needs. It can be understood that the type and number of lenses in the second optical component 206B are variable, and the specific type and number of lenses used can be designed according to actual needs.

In some embodiments, as shown in Fig. 2C, the first camera 208A and the second camera 208B may be symmetrically arranged in the first lens barrel 202A relative to the axis of the first lens barrel 202A. In this way, the human eye images of the first eye 1022A collected by the first camera 208A and the second camera 208B may be symmetrical, and the processing efficiency of the subsequent algorithm may be further improved.

Similarly, in some embodiments, as shown in FIG2C , the third camera 208C and the fourth camera 208D are symmetrically arranged in the second lens barrel 202B relative to the axis of the second lens barrel 202B. In this way, the human eye images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can also be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

In some embodiments, as shown in FIG2C , the first camera 208A and the second camera 208B are both oriented toward the light-emitting side of the first display module 204A, and the first camera 208A and the second camera 208B are oriented at the same angle to the axis of the first lens barrel 202A, for example, the angles are both β. In this way, the human eye images of the first eye 1022A captured by the first camera 208A and the second camera 208B can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

Similarly, in some embodiments, as shown in FIG2C , the third camera 208C and the fourth camera 208D are both facing the light-emitting side of the second display module 204B, and the angles between the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, the angles are both β. In this way, the human eye images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

FIG. 2D shows a schematic diagram of another exemplary wearable device 200 provided by an embodiment of the present disclosure.

In some embodiments, as shown in FIG2D , the first camera 208A and the second camera 208B are both facing the first display module 204A, and the first reflection structure 210A corresponding to the first camera 208A and the second reflection structure 210B corresponding to the second camera 208B are provided in the first lens barrel 202A. The first reflection structure 210A and the second reflection structure 210B may be structures with reflection surfaces such as reflection films or reflection mirrors. The first reflection structure 210A and the second reflection structure 210B may correspond to the first camera 208A and the second camera 208B respectively. The first reflection structure 210A is used to reflect the light from the first eye 1022A to the first camera 208A, and the second reflection structure 210B is used to reflect the light from the first eye 1022A to the second camera 208B. Through the reflection of the light by the first reflection structure 210A and the second reflection structure 210B, the first camera 208A and the second camera 208B can still collect the human eye image of the first eye 1022A. Furthermore, since a reflection process is added to the optical path, the observation angle γ is further reduced, and the first camera 208A and the second camera 208B can better image the first eye 1022A.

Similarly, in some embodiments, as shown in FIG. 2D , the third camera 208C and the fourth camera 208D are both facing the second display module 204B, and the second lens barrel 202B is provided with a third reflective structure 210C corresponding to the third camera 208C and a fourth reflective structure 210D corresponding to the fourth camera 208D, and the third reflective structure 210C and the fourth reflective structure 210D can be structures with a reflective surface such as a reflective film or a reflective mirror. The third reflective structure 210C and the fourth reflective structure 210D can correspond to the third camera 208C and the fourth camera 208D respectively, and the third reflective structure 210C is used to reflect the light from the second eye 1022B to the third camera 208C, and the fourth reflective structure 210D is used to reflect the light from the second eye 1022B to the fourth camera 208D. The reflection of the light by the third reflective structure 210C and the fourth reflective structure 210D enables the third camera 208C and the fourth camera 208D to still capture the human eye image of the second eye 1022B. In addition, because a reflection process is added to the optical path, the observation angle γ is further reduced, and the third camera 208C and the fourth camera 208D can better image the second eye 1022B.

In some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or line of sight tracking, as shown in FIG2D , the first camera 208A and the second camera 208B may be symmetrically arranged in the first lens barrel 202A relative to the axis of the first lens barrel 202A, and the first reflection structure 210A and the second reflection structure 210B may also be symmetrically arranged in the first lens barrel 202A relative to the axis of the first lens barrel 202A. In this way, the human eye images of the first eye 1022A collected by the first camera 208A and the second camera 208B may be symmetrical, and the processing efficiency of the subsequent algorithms may be further improved.

Similarly, in some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or line of sight tracking, as shown in FIG2D , the third camera 208C and the fourth camera 208D may be symmetrically arranged in the second lens barrel 202B relative to the axis of the second lens barrel 202B, and the third reflection structure 210C and the fourth reflection structure 210D may also be symmetrically arranged in the second lens barrel 202B relative to the axis of the second lens barrel 202B. In this way, the human eye images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D may be symmetrical, and the processing efficiency of the subsequent algorithms may be further improved.

In some embodiments, as shown in FIG2D , the first camera 208A and the second camera 208B are both facing the first display module 204A, and the angles between the first camera 208A and the second camera 208B and the axis of the first lens barrel 202A are equal, for example, the angles are both γ; at the same time, the angles between the first reflective structure 210A and the second reflective structure 210B and the axis of the first lens barrel 202A are also equal. In this way, the human eye images of the first eye 1022A collected by the first camera 208A and the second camera 208B can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

In some embodiments, as shown in FIG2D , the third camera 208C and the fourth camera 208D are both facing the second display module 204B and the angles between the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, the angles are both γ; at the same time, the angles between the third reflective structure 210C and the fourth reflective structure 210D and the axis of the second lens barrel 202B are also equal. In this way, the human eye images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be strictly symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

The above embodiments are all described by setting two cameras in two lens barrels respectively. It can be understood that one of the two lens barrels can be set with two or more cameras, and the other lens barrel can be set with one camera. In this case, the algorithm accuracy can still be improved to a certain extent, which should also fall within the protection scope of the present disclosure. In addition, the structures shown in Figures 2A and 2B can also be matched in the same wearable device 200. For example, the lens barrel corresponding to the first eye has the structure shown in Figure 2A, and the lens barrel corresponding to the second eye has the structure shown in Figure 2B. Of course, the structures corresponding to the first eye and the second eye can also be swapped.

It can be seen from the above embodiments that the embodiments of the present disclosure can obtain better imaging quality and improve the comfort of the wearable device 200 by setting at least two cameras 208 inside at least one lens barrel of the wearable device 200. In some embodiments, by placing two cameras in each of the left and right lens barrels of the wearable device 200, not only can the aforementioned adverse effects caused by the cameras outside the lens barrel be avoided, but the two cameras can provide more observation information and are less affected by occlusion, so the accuracy of algorithms such as pupil distance estimation or line of sight tracking can be improved.

Furthermore, compared with placing the camera outside the lens barrel, placing the camera inside the lens barrel will affect the camera imaging not only by the camera module itself but also by the optical components in the lens barrel.

As shown in FIGS. 2A and 2B , relative to the camera 1048 of FIG. 1B , in the optical path from the cameras 208A and 208B to the human eye 1022, there is an optical component 206 or a portion of the optical component 206 in the lens barrel (depending on the relative position relationship between the cameras 208A and 208B and the lenses in the optical component 206, for example, in addition to the positions shown in FIGS. 2A and 2B , the cameras 208A and 208B may also be arranged between the lens 2062 and the lens 2064), so that when the cameras 208A and 208B image the human eye 1022, the optical component 206 or a portion of the optical component 206 affects or changes the optical path from the cameras 208A and 208B to the human eye 1022, resulting in that the distortion of the images obtained by the imaging of the cameras 208A and 208B may no longer be symmetrical.

Furthermore, since the optical paths from the cameras 208A and 208B to the human eye 1022 are changed, the cameras 208A and 208B do not have a unified projection center, which makes it impossible to use a parameterized camera model to fit the projection process of the cameras in the lens barrel to process the distortion, making it difficult to calibrate the camera parameters.

In addition, in some cases, as shown in FIG. 2C and FIG. 2D , since different cameras are set at different positions in the wearable device 200, it is also necessary to calibrate the posture relationship between different cameras, and such calibration is also difficult to implement. In particular, when different cameras are located in different lens barrels (for example, between the first camera 208A and the third camera 208C or the fourth camera 208D, or between the second camera 208B and the third camera 208C or the fourth camera 208D), such calibration will become more complicated.

In view of this, the embodiments of the present disclosure also provide a camera parameter calibration method, which can use a non-parametric camera model to implement parameter calibration of the camera in the lens barrel. In some embodiments, for cameras in different lens barrels, a camera extrinsic calibration algorithm based on non-common viewpoints is proposed, thereby solving the problem of difficult calibration of cameras in different lens barrels.

FIG. 3A is a schematic flow chart of an exemplary method 300 provided in an embodiment of the present disclosure.

The method 300 can be applied to any computer device with data processing capabilities, and can be used to calibrate the parameters of the camera 208A, 208B, 208C or 208D in the wearable device 200 shown in Figure 2A, Figure 2B, Figure 2C or Figure 2D. As shown in Figure 3A, the method 300 can further include the following steps.

In step 302, multiple images obtained by photographing a calibration reference object by at least two cameras may be received. In this step, the multiple images may be images captured by different cameras. As an optional embodiment, in order to implement calibration of a first parameter (e.g., an intrinsic parameter) and a second parameter (e.g., an extrinsic parameter) of a camera, multiple images may be captured for intrinsic parameter calibration and multiple images may be captured for extrinsic parameter calibration, respectively.

FIG4A shows a schematic diagram of an exemplary image collection scenario 400 according to an embodiment of the present disclosure.

As shown in FIG4A , in the image acquisition scene 400, a calibration reference object 402 may be set at a fixed position. In some embodiments, the calibration reference object 402 may be a calibration plate or a chart, and some black and white checkerboards are set on the calibration reference object 402. When the camera captures the calibration reference object 402 to obtain a corresponding image, the corresponding camera parameters may be calculated according to the correspondence between the checkerboards in the image and the checkerboards of the calibration reference object 402.

When capturing images (or capturing images) of the calibration reference object 402, the camera to be calibrated (e.g., camera 208A) can be controlled to move along a certain motion trajectory, and the calibration reference object 402 can be continuously captured during the movement of the camera, thereby obtaining multiple images captured by the camera in different postures, which are used as data for subsequent calculation of camera parameters. Since the captured images are obtained by the camera performing image capture in multiple postures, the camera parameters calculated subsequently are more robust and can be applied to a wider range.

In some embodiments, if the at least two cameras provided in the wearable device are cameras of different types or cameras with different camera parameters, multiple images need to be collected in the above manner for each camera or camera with each parameter, respectively, for calibrating the first parameter (e.g., internal parameter) of the corresponding camera. If the camera parameters of the at least two cameras provided in the wearable device are consistent, a camera can be selected to implement the aforementioned image acquisition operation, so as to calibrate the internal parameters of the camera, and the calibrated internal parameters can be applied to other cameras in the wearable device.

FIG4B shows a schematic diagram of another exemplary image collection scenario 410 according to an embodiment of the present disclosure.

As shown in FIG. 4B , similar to the image collection scene 400 , in the image collection scene 410 , a calibration reference object 402 may also be set at a fixed position.

Different from the image collection scene 400, in order to calculate the pose relationship (second parameter) between at least two cameras in the wearable device 200, the at least two cameras can be installed in the lens barrel of the wearable device 200 and the wearable device 200 can be fully assembled. The assembled wearable device 200 or its prototype can be used to perform external parameter image collection, and then the corresponding external parameters (for example, the pose relationship between different cameras) can be calculated based on the collected images.

As shown in FIG. 4B , taking the case where two cameras are respectively arranged in the two lens barrels of the wearable device 200, when capturing images (or capturing images) of the calibration reference object 402, the wearable device 200 can be controlled to move along a certain motion trajectory, and the calibration reference object 402 can be continuously captured by four cameras during the movement of the wearable device 200, so that when the relative posture relationship of the four cameras remains unchanged, multiple images captured by the four cameras in different postures are obtained as image data for subsequent calculation of the second parameter. Since the captured images are obtained by the camera capturing images in multiple postures, the camera parameters calculated subsequently have better robustness and can be applied to a wider range.

Based on the above content, it can be understood that in the image acquisition scene 410, each camera can capture multiple images. In addition to being used to calculate the relative posture relationship of the four cameras, the images captured by each camera can also be used to calculate its own internal parameters. Therefore, in some embodiments, only the image acquisition scene 410 can be used to capture images, and the first parameter and the second parameter of the camera can be calibrated based on these images.

In step 302, the computer device may receive a plurality of images captured in the aforementioned scene 400 and/or scene 410 for subsequent processing.

After acquiring the required multiple images, the camera can be calibrated based on the multiple images. The parameter calibration can include calibrating a first parameter and a second parameter, wherein the first parameter can be an intrinsic parameter of the camera and the second parameter can be an extrinsic parameter of the camera.

In some embodiments, the intrinsic parameters of the camera may be calibrated first. However, as mentioned above, since the camera is set in the lens barrel, the camera imaging is not only affected by the module itself, but also by the lens effect of the lens barrel, so that the imaging distortion is no longer symmetrical about the image principal point, and the camera does not have a unified projection center, which makes it impossible to use a parametric camera model to fit the projection process of the camera in the lens barrel to handle the asymmetric distortion. Therefore, in some embodiments, a non-parametric camera model is provided to calibrate the intrinsic parameters of the camera.

Therefore, the projection relationship between the pixels of the multiple images and the calibration reference object may be determined in step 304. In this step, the projection relationship between the image and the calibration reference object 402 may be established based on the acquired image, thereby establishing a corresponding relationship between the pixel and the projection direction (pixel-ray).

It can be understood that the projection relationship between the pixel points of the image and the calibration reference object represents the intrinsic parameters (i.e., the internal parameters) of the camera. Therefore, when calculating the projection relationship for a specific camera (e.g., camera 208A), it is necessary to use the images captured by the specific camera to establish the projection relationship. Therefore, taking the image capture scene shown in FIG. 4B as an example, it is necessary to distinguish among the multiple images the multiple first images captured by the first camera 208A, the multiple second images captured by the second camera 208B, the multiple third images captured by the third camera 208C, and the multiple fourth images captured by the fourth camera 208D.

The following takes the first camera 208A as an example to perform projection relationship calculation.

In some embodiments, as shown in FIG. 3B , step 304 of determining the projection relationship between the pixels of the plurality of images and the calibration reference object may further include the following steps:

In step 3042, a first number of target images are selected from the plurality of images.

In this step, a projection relationship can be first established by selecting a certain number of target images from the multiple first images (images captured by the first camera 208A) in the multiple images, and the remaining first images can be used to supplement the parts for which the projection relationship is not established.

It can be understood that the first number is not specifically limited as long as it is sufficient to proceed with the subsequent steps. As an optional embodiment, the first number can be 3. FIG. 4C shows three selected target images 412, 414, and 416. As shown in FIG. 4C, the target images 412, 414, and 416 respectively show images obtained by the first camera 208A taking the calibration reference object 402 in different postures. It can be understood that by calibrating the camera parameters of the images captured in different camera postures, the calculated camera parameters can be made more robust.

In step 3044, at least one local area is selected in each of the target images (for example, the area corresponding to a grid in the chessboard can be considered as a local area), and a homography transformation matrix between each of the local areas and the calibration reference object is established.

After selecting at least one local area of each target image, a homography transformation matrix with the calibration reference object 402 can be constructed for each local area of each target image to establish a corresponding relationship (pixel-coordinate) between each pixel point contained in the local area and the coordinate system of the calibration reference object 402.

It can be known that the coordinate system of the position of the calibration reference object 402 is known, and the checkerboard on the calibration reference object 402 corresponds to the checkerboard image in the target image. At the same time, the coordinates of the four vertices corresponding to the local area in the camera coordinate system of the target image are also known. Based on these known information, the homography transformation matrix H between the local area and the calibration reference object 402 can be obtained.

In this way, a corresponding relationship is established between the pixel points included in the local area and the calibration reference object 402 or the coordinate system where the calibration reference object 402 is located.

In step 3046, the projection relationship between the pixel points included in the local area and the calibration reference object is determined according to the homography transformation matrix.

FIG4D is a schematic diagram showing a projection relationship between a pixel point and a calibration reference object according to an embodiment of the present disclosure.

As shown in Figure 4D, the area corresponding to the four vertices connected by the four dotted lines in the figure is the local area. By constructing the homography transformation matrix, the homography transformation matrix can be used to calculate the projection relationship between each pixel point contained in the local area in the target image and the corresponding point on the calibration reference object.

In step 3048, the first number of target images are converted into a reference coordinate system, and the projection relationship between the pixel points of the multiple images and the calibration reference object is determined based on the area with the established projection relationship corresponding to the local area.

As shown in FIG4C , the three target images 412 , 414 , and 416 are acquired under different camera postures. When constructing the homography transformation matrix of the local area, it is constructed based on the camera coordinate system corresponding to each target image. It can be understood that in order to unify the projection relationship obtained by each target image under the same reference coordinate system, in this step, the three target images 412 , 414 , and 416 can be converted to the reference coordinate system.

After completing the processing of the three target images 412, 414, and 416 according to the above steps, a method similar to the previous one can be used to continue selecting local areas of new target images in the remaining first image to construct a homography transformation matrix until the projection relationship between each pixel point in the entire image and the calibration reference object is completed.

According to the above embodiments, it can be known that in some embodiments, after all image processing is completed and the entire image is calibrated, some pixels may correspond to multiple projection direction data. Therefore, these data can be averaged to obtain a projection direction as the projection direction of the pixel to obtain better projection direction data. In addition, only the projection direction after the average processing can be stored in subsequent storage, thereby saving storage space.

It can be understood that the aforementioned method can be used to establish the projection relationship between the image pixels and the calibration reference object 402 for the second camera 208B, the third camera 208C, the fourth camera 208D and possibly more cameras (depending on the number of cameras set in the wearable device), and will not be repeated here.

After the projection relationship is established, the first parameter set and the second parameter set of the at least two cameras may be determined according to the projection relationship in step 306. The first parameter set includes a plurality of first parameters, which are used to characterize the target pixel points in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one set of second parameters, which are used to indicate the posture relationship between the at least two cameras.

In some embodiments, a first parameter set of the at least two cameras may be determined first according to the projection relationship, and then a second parameter set may be determined.

FIG. 4E shows a schematic diagram of an exemplary first parameter according to an embodiment of the present disclosure.

As shown in FIG. 4E , when constructing the aforementioned projection relationship, at least some of the pixels in the image under the reference coordinate system correspond to some points on the calibration reference object one by one, so that a straight line equation about the pixel can be obtained, and the straight line represented by the straight line equation passes through the corresponding pixel and has a ray (ray) as shown in the figure to represent its projection direction. Such a combination of pixel points and projection directions can be used as a first parameter (the camera parameter can be considered as an internal parameter of the camera), and the camera parameter is used to realize the conversion from the coordinate system of the image captured by the camera to the camera coordinate system, so that it can be used to calculate the position information of certain features (for example, pupil) in the image captured by the camera in three-dimensional space (camera coordinate system). In addition, since multiple local areas of the first image are used to respectively establish homography transformation matrices with the calibration reference object during the calculation of the camera parameters, when the camera parameters are used to perform coordinate system conversion calculations, the projection direction corresponding to the pixel point obtained already includes the correction of the image distortion at the position, and no additional distortion correction is required.

In some embodiments, the previously obtained data may be simplified and used as the first parameters of the camera, thereby saving space for storing the first parameters. For example, a spline surface may be used to fit the initial set of first parameters obtained by the aforementioned calibration, and then the control points of the fitted spline surface may be optimized using a bundle adjustment (BA) algorithm, and the first parameters corresponding to all the control points of the optimized spline surface may be used as the first parameter set of the at least two cameras.

When the first parameters need to be used, a complete set of first parameters can be obtained through an interpolation algorithm based on the fitting spline surface and the optimized control points.

It can be seen that the above method only provides a calculation method for the first parameter of a single camera. For each camera in the wearable device 200, the above method can be used to calculate the corresponding first parameter, which will not be repeated here.

After obtaining the first parameter set, the second parameter set may be further calibrated.

Returning to FIG. 2C , the wearable device 200 includes a first camera 208A and a second camera 208B disposed in the first lens barrel 202A for collecting human eye images of the first eye 1022A, and a third camera 208C and a fourth camera 208D disposed in the second lens 202B for collecting human eye images of the second eye 1022B. Since the four cameras are located at different positions in the wearable device 200, in order to determine the relative relationship of the images collected by the four cameras, it is necessary to know the position relationship between the four cameras as the second parameter, thereby completing the calibration of the camera extrinsic parameters.

The following takes the calibration of four cameras in the wearable device 200 as an example to illustrate how to calculate the second parameter.

As shown in FIG2C , since the two cameras arranged in the same lens barrel of the wearable device 200 need to capture images of the same human eye, generally, the two cameras have a common viewpoint (or an intersection of optical paths). Therefore, a binocular positioning algorithm can be used to obtain the posture relationship between the two cameras in the same lens barrel.

Therefore, in some embodiments, the second parameter may include the pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D, and the pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera are calculated according to the projection relationship based on a binocular positioning algorithm.

Optionally, step 306 of determining the first parameter set and the second parameter set of the at least two cameras according to the projection relationship may further include: determining the posture relationship between the first camera 208A and the second camera 208B and the posture relationship between the third camera 208C and the fourth camera 208D according to the projection relationship using a binocular positioning algorithm.

2C, it can be understood that, because the cameras in different lens barrels collect images of different human eyes, the cameras in different lens barrels may not have a common viewpoint, and the binocular positioning algorithm cannot be used to calculate the pose relationship, or the pose relationship calculated by the binocular positioning algorithm may be inaccurate. Therefore, an embodiment of the present disclosure provides a method for calculating the pose relationship between cameras in different lens barrels.

FIG. 4F shows a schematic diagram of a calibration principle of a second parameter of an exemplary camera in a different-side lens barrel according to an embodiment of the present disclosure.

[Corrected 15.01.2025 in accordance with Article 91]
As shown in FIG4F , taking the calculation of the pose relationship between the first camera 208A and the fourth camera 208D as an example, the two cameras have no common viewpoint. When the two cameras move to a certain position, the relative positions of the two cameras and the calibration reference object 402 are shown in FIG4F . The pose relationship of the first camera 208A relative to the calibration reference object 402 is T _1,i , the pose relationship of the fourth camera 208D relative to the calibration reference object 402 is T _4,i , and the pose relationship between the first camera 208A and the fourth camera 208D is T ₁₄ . Then, using the observation data of the first camera 208A and the fourth camera 208D (the first image set and the fourth image set obtained in the mapping step), a BA optimization problem can be constructed, and the pose relationship T ₁₄ can be obtained by solving the problem.

Therefore, in some embodiments, the second parameter may also include a pose relationship between the first camera and the third camera, a pose relationship between the first camera and the fourth camera, a pose relationship between the second camera and the third camera, and a pose relationship between the second camera and the fourth camera.

Optionally, step 306 of determining the first parameter set and the second parameter set of the at least two cameras according to the projection relationship may further include: determining the posture relationship between the first camera and the third camera, the posture relationship between the first camera and the fourth camera, the posture relationship between the second camera and the third camera, and the posture relationship between the second camera and the fourth camera according to the multiple images and the projection relationship.

Specifically, an optimization function may be constructed first.

Optionally, the optimization function may include a first formula for representing the error between a detection point in the first image captured by the first camera 208A and a spatial position corresponding to the detection point (a corresponding position on the calibration reference object 402) projected to a two-dimensional point in the image coordinate system using a first parameter of the first camera 208A corresponding to the detection point. Optionally, the first formula is expressed as:

f _cam1,1 =π1(T _1,i ,P _cam1 )-d _cam1

Wherein, π1 is the projection function corresponding to the first camera 208A (i.e., the projection relationship between the pixel points obtained previously and the calibration reference object 402), T _1,i is the external parameter of the first camera 208A and the calibration reference object 402 corresponding to the i-th first image (which can be calculated based on the spatial coordinates of the first image and the calibration reference object 402), P _cam1 is the 3D point corresponding to the i-th first image of the first camera 208A on the calibration reference object 402 (i.e., the spatial coordinates of the points corresponding to each pixel point of the first image on the calibration reference object 402), and d _cam1 is all the detection points corresponding to the first camera 208A.

Optionally, the optimization function may further include a second formula for expressing the error between a detection point in the fourth image captured by the fourth camera 208D and a spatial position corresponding to the detection point (a corresponding position on the calibration reference object 402) projected to a two-dimensional point in the image coordinate system using the first parameter of the fourth camera 208A corresponding to the detection point. Optionally, the second formula is expressed as:

f _cam4,2 =π4(T _4,i ,P _cam4 )-d _cam4

Wherein, π4 is the projection function corresponding to the fourth camera 208D (that is, the projection relationship between the pixel points obtained previously and the calibration reference object 402), T _4,i is the external parameter of the fourth camera 208D and the calibration reference object 402 corresponding to the i-th fourth image (which can be calculated according to the coordinate information of the fourth image and the calibration reference object 402), P _cam4 is the 3D point corresponding to the i-th fourth image of the fourth camera 208D on the calibration reference object 402 (that is, the spatial coordinates of the points corresponding to each pixel point of the fourth image on the calibration reference object 402), and d _cam4 is all the detection points corresponding to the fourth camera 208D.

Optionally, the optimization function may further include a third formula for representing the error between a detection point in the fourth image captured by the fourth camera 208D and a spatial position corresponding to the detection point (a corresponding position on the calibration reference object 402) projected to a two-dimensional point in the image coordinate system using the first parameter of the first camera 208A corresponding to the detection point and the posture relationship between the first camera and the fourth camera. Optionally, the third formula is expressed as:

f _cam4,1 =π4(T ₁₄ *T _1,i ,P _cam4 )-d _cam4

Wherein, _T14 is the posture relationship between the first camera 208A and the fourth camera 208D, that is, the external parameters of the first camera and the fourth camera required to be obtained in the embodiment of the present disclosure.

Optionally, the optimization function may further include a fourth formula for representing the error between a detection point in the fourth image captured by the fourth camera 208D and a spatial position corresponding to the detection point (a corresponding position on the calibration reference object 402) projected to a two-dimensional point in the image coordinate system using the first parameter of the fourth camera corresponding to the detection point and the posture relationship between the first camera and the fourth camera. Optionally, the fourth formula is expressed as:

in, is the transposed matrix of the position and posture relationship between the first camera 208A and the fourth camera 208D, which is obtained by transposing _T14 .

In some embodiments, the detection points d _cam1 and d _cam4 may be calculated in the following manner.

FIG. 4G is a schematic diagram showing an image obtained after the captured image is binarized in an embodiment of the present disclosure.

As shown in FIG4G , after binarization, the pixels on the image are either black or white. Through feature detection, the vertices of the checkerboard grid can be identified. These identifiable vertices can be used as detection points d. The detection points detected on the i-th image can be represented as d _i .

Optionally, the optimization function may further determine an error function based on the first formula, the second formula, the third formula, and the fourth formula. Optionally, the error function is expressed as:

f＝ _fcam1,1 + _fcam4,1 + _fcam1,2 + _fcam4,2

Finally, optionally, an optimization function may be constructed based on the error function to determine the final external parameters of the first camera 208A and the fourth camera 208D. Optionally, the optimization function is expressed as follows:

Among them, ρ is the loss function, f is the error function O _i is all the information contained in the i-th image, and I is the number of input images.

It can be seen that the loss function res(π, T) is a function of posture. By solving the loss function using the Levenberg-Marquardt method (LM for short), the optimal solution obtained is the final posture relationship T ₁₄ .

It can be understood that the above method can be used to obtain the posture relationship of the two cameras in the different side lens barrels, and will not be repeated here.

Considering that there are many permutations and combinations of cameras on opposite sides, if each pair of cameras uses the above method to calculate the pose relationship, the amount of calculation will increase. Therefore, in some embodiments, according to the multiple images and the projection relationship, the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera include: determining the pose relationship between the first camera and the fourth camera according to the multiple images and the projection relationship; determining the pose relationship between the first camera and the third camera according to the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera; determining the pose relationship between the second camera and the third camera according to the pose relationship between the first camera and the second camera and the pose relationship between the first camera and the third camera; determining the pose relationship between the second camera and the fourth camera according to the pose relationship between the third camera and the fourth camera and the pose relationship between the second camera and the third camera.

In this way, after calculating the posture relationship between the first camera 208A and the fourth camera 208D, based on the calculated posture relationship between the first camera 208A and the second camera 208B and the posture relationship between the third camera 208C and the fourth camera 208D, the posture relationships of other arrangements and combinations can be obtained through data conversion, thereby saving the amount of calculation.

It can be understood that the above method provides a calculation method that can be used when there is no common viewpoint between the two cameras in the different-side barrels. However, when there is a common viewpoint between the two cameras in the different-side barrels, the binocular positioning algorithm can still be used to calculate the pose relationship.

It should be noted that the above example only takes the example of setting two cameras in the two lens barrels of the wearable device 200 respectively. It can be understood that the number of cameras in the lens barrel can be less or more according to actual needs. However, in any case, based on the inventive concept of the embodiment provided by the present disclosure, the corresponding camera parameters can still be calculated, which will not be repeated here.

In a more specific embodiment, the camera parameter calibration method provided by the embodiment of the present disclosure may include the steps of image acquisition, internal parameter calibration, external parameter calibration, and optimizing and saving parameters, so as to obtain better camera parameters for subsequent algorithm calculation.

It can be seen from the above embodiments that the camera parameter calibration method provided by the embodiments of the present disclosure provides a feasible calibration scheme for the intrinsic and extrinsic parameters of the camera for scenes where the imaging distortion of the camera in the lens barrel is asymmetric and there is no unified projection center.

It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The embodiment of the present disclosure also provides a wearable device.

FIG. 5 shows a schematic diagram of an exemplary wearable device 500 provided by an embodiment of the present disclosure.

As shown in FIG. 5 , similar to FIG. 2A and FIG. 2B , the wearable device 500 includes a lens barrel 202 , in which a display module 204 , cameras 208A, 208B, and an optical component 206 are disposed. The cameras 208A, 208B and the optical component 206 are located on the light emitting side of the display module 204 , and the cameras 208A, 208B are located between the optical component 206 and the display module 204 .

In some embodiments, the shooting direction of the cameras 208A and 208B is toward the optical assembly 206, as shown in FIG2A. In other embodiments, as shown in FIG2B, a reflective structure 210A and 210B is disposed between the display module 204 and the cameras 208A and 208B, the reflective surface of the reflective structure 210A and 210B is toward the light emitting side of the display module 204, and the shooting direction of the cameras 208A and 208B is toward the reflective structure 210A and 210B.

Further, as shown in Figure 5, the wearable device 500 also includes a processing module 502, which is electrically coupled to the camera 208 and the display module 204, respectively, and is configured to: obtain the first parameter set and the second parameter set obtained by the aforementioned method 300 and the images taken by the cameras 208A and 208B; use the first parameter set and the second parameter set and the image to solve the position of the target area in the image in space.

As mentioned above, the first parameter set includes camera parameters in combination of pixel points and projection directions (the camera parameters can be considered as the internal parameters of the camera), and the camera parameters are used to realize the conversion from the coordinate system of the image captured by the camera to the camera coordinate system, so that it can be used to calculate the position information of certain features (for example, pupil) in the image captured by the camera in the three-dimensional space (camera coordinate system). In addition, since the calculation process of the camera parameters includes the correction of image distortion, when the camera parameters are used for coordinate system conversion calculation, the distortion can be directly corrected without the need for additional distortion correction.

The second parameter set includes a second parameter representing the position and posture relationship between different cameras, and the image information collected by different cameras can be unified (for example, unified into the same camera coordinate system) according to the second parameter.

The wearable device provided by the embodiment of the present disclosure sets the camera inside the lens barrel, which is less affected by occlusion, and can improve the accuracy of pupil distance estimation or line of sight tracking algorithm. Furthermore, by setting the camera to at least two, more observation information can be provided, further improving the accuracy of the algorithm. The camera parameter calibration method and related equipment provided by the embodiment of the present disclosure provide a feasible camera parameter calibration solution for scenes where the camera imaging distortion in the lens barrel is asymmetric and there is no unified projection center.

The embodiment of the present disclosure also provides a computer device for implementing the above-mentioned method 300. FIG6 shows a schematic diagram of the hardware structure of an exemplary computer device 600 provided in the embodiment of the present disclosure. The computer device 600 can be used to implement the head-mounted wearable device 104 of FIG1A, the wearable device 200 of FIG2A to FIG2D, and can also be used to implement the external device 112 of FIG1A, and can also be used to implement the server 114 of FIG1A. In some scenarios, the computer device 600 can also be used to implement the database server 116 of FIG1A.

As shown in Fig. 6, computer device 600 may include: processor 602, memory 604, network module 606, peripheral interface 608 and bus 610. Processor 602, memory 604, network module 606 and peripheral interface 608 are connected to each other in communication within computer device 600 via bus 610.

Processor 602 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 may be used to perform functions related to the technology described in the present disclosure. In some embodiments, processor 602 may also include multiple processors integrated into a single logical component. For example, as shown in FIG. 6 , processor 602 may include multiple processors 602a, 602b, and 602c.

The memory 604 may be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 6, the data stored in the memory 604 may include program instructions (e.g., program instructions for implementing the method 300 or 500 of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 602 may also access the program instructions and data stored in the memory 604, and execute the program instructions to operate on the data to be processed. The memory 604 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 604 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

The network interface 606 can be configured to provide the computer device 600 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the above specific examples.

The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

The bus 610 may be configured to transmit information between various components of the computer device 600 (e.g., the processor 602, the memory 604, the network interface 606, and the peripheral interface 608), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

It should be noted that, although the architecture of the above-mentioned computer device 600 only shows the processor 602, the memory 604, the network interface 606, the peripheral interface 608 and the bus 610, in the specific implementation process, the architecture of the computer device 600 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the above-mentioned computer device 600 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

The embodiment of the present disclosure also provides a camera parameter calibration device. Figure 7 shows a schematic diagram of an exemplary device 700 provided by the embodiment of the present disclosure. As shown in Figure 7, the device 700 can be used to implement the method 300, and can further include the following modules.

The receiving module 702 is configured to: receive multiple images obtained by photographing a calibrated reference object by at least two cameras; the wearable device also includes a binocular display module, the lens barrel is arranged on the light-emitting side of the binocular display module, an optical component is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical component; the first determination module 704 is configured to: determine the projection relationship between the pixel points of the multiple images and the calibrated reference object; the second determination module 706 is configured to: determine the first parameter set and the second parameter set of the at least two cameras according to the projection relationship.

Among them, the first parameter set includes multiple first parameters, the first parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one group of second parameters, and the at least one group of second parameters is used to indicate the posture relationship between the at least two cameras.

In some embodiments, the at least two cameras include a first camera and a second camera for capturing a first eye image of a human eye and a third camera and a fourth camera for capturing a second eye image of a human eye, the first camera and the second camera are arranged in a first lens barrel of the wearable device, and the third camera and the fourth camera are arranged in a second lens barrel of the wearable device; the second parameter includes a posture relationship between the first camera and the second camera and a posture relationship between the third camera and the fourth camera, and the posture relationship between the first camera and the second camera and the posture relationship between the third camera and the fourth camera are calculated based on a binocular positioning algorithm according to the projection relationship.

In some embodiments, the second parameters also include a posture relationship between the first camera and the third camera, a posture relationship between the first camera and the fourth camera, a posture relationship between the second camera and the third camera, and a posture relationship between the second camera and the fourth camera.

In some embodiments, the second determination module 706 is configured to: determine the posture relationship between the first camera and the fourth camera based on the multiple images and the projection relationship; determine the posture relationship between the first camera and the third camera based on the posture relationship between the third camera and the fourth camera and the posture relationship between the first camera and the fourth camera; determine the posture relationship between the second camera and the third camera based on the posture relationship between the first camera and the second camera and the posture relationship between the first camera and the third camera; determine the posture relationship between the second camera and the fourth camera based on the posture relationship between the third camera and the fourth camera and the posture relationship between the second camera and the third camera.

In some embodiments, the first determination module 704 is configured to: select a first number of target images from the multiple images; select at least one local area in each of the target images, and establish a homography transformation matrix between each of the local areas and the calibrated reference object; determine the projection relationship between the pixel points contained in the local area and the calibrated reference object according to the homography transformation matrix; convert the first number of target images to a reference coordinate system, and determine the projection relationship between the pixel points of the multiple images and the calibrated reference object according to the area with the established projection relationship corresponding to the local area.

In some embodiments, the second determination module 706 is configured to: determine an initial first parameter set of the at least two cameras according to the projection relationship, the initial first parameter set including multiple first parameters corresponding one-to-one to multiple pixel points of the image; fit the initial first parameter set using a spline surface to obtain a fitted spline surface; and optimize the control points of the fitted spline surface based on a bundle adjustment algorithm to obtain the first parameter set of the at least two cameras.

For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and/or hardware.

The device of the above embodiment is used to implement the corresponding method 300 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 300 described in any of the above embodiments.

The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method 300 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same inventive concept, corresponding to the method 300 of any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and/or the processor execute the method 300. Corresponding to the execution subject corresponding to each step in each embodiment of the method 300, the processor that executes the corresponding step can belong to the corresponding execution subject.

The computer program product of the above embodiment is used to enable the computer and/or the processor to execute the method 300 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power/ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (15)

A camera parameter calibration method, comprising: Receiving a plurality of images obtained by photographing a calibration reference object by at least two cameras, wherein the at least two cameras are arranged in a lens barrel of a wearable device; the wearable device further comprises a binocular display module, the lens barrel is arranged on a light-emitting side of the binocular display module, an optical component is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical component; Determining a projection relationship between pixel points of the plurality of images and the calibration reference object; Determining a first parameter set and a second parameter set of the at least two cameras according to the projection relationship; Among them, the first parameter set includes multiple first parameters, the first parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one group of second parameters, and the at least one group of second parameters is used to indicate the posture relationship between the at least two cameras. The method of claim 1, wherein the at least two cameras include a first camera and a second camera for acquiring a human eye image of a first eye and a third camera and a fourth camera for acquiring a human eye image of a second eye, the first camera and the second camera are arranged in a first lens barrel of the wearable device, and the third camera and the fourth camera are arranged in a second lens barrel of the wearable device; The second parameter includes a posture relationship between the first camera and the second camera and a posture relationship between the third camera and the fourth camera, and the posture relationship between the first camera and the second camera and the posture relationship between the third camera and the fourth camera are calculated based on the projection relationship based on a binocular positioning algorithm. The method as claimed in claim 2, wherein the second parameter further includes a pose relationship between the first camera and the third camera, a pose relationship between the first camera and the fourth camera, a pose relationship between the second camera and the third camera, and a pose relationship between the second camera and the fourth camera. The method of claim 3, wherein determining the first parameter set and the second parameter set of the at least two cameras according to the projection relationship comprises: Determining a posture relationship between the first camera and the fourth camera according to the multiple images and the projection relationship; Determine a posture relationship between the first camera and the third camera according to a posture relationship between the third camera and the fourth camera and a posture relationship between the first camera and the fourth camera; Determine a posture relationship between the second camera and the third camera according to a posture relationship between the first camera and the second camera and a posture relationship between the first camera and the third camera; The position and posture relationship between the second camera and the fourth camera is determined according to the position and posture relationship between the third camera and the fourth camera and the position and posture relationship between the second camera and the third camera. The method of claim 1, wherein determining the projection relationship between the pixels of the plurality of images and the calibration reference object comprises: Selecting a first number of target images from the plurality of images; Selecting at least one local area in each of the target images, and establishing a homography transformation matrix between each of the local areas and the calibration reference object; Determining a projection relationship between pixel points included in the local area and the calibration reference object according to the homography transformation matrix; The first number of target images are converted into a reference coordinate system, and the projection relationship between the pixel points of the plurality of images and the calibration reference object is determined according to the area with the established projection relationship corresponding to the local area. The method of claim 1, wherein determining the first parameter set and the second parameter set of the at least two cameras according to the projection relationship comprises: Determine, according to the projection relationship, an initial first parameter set of the at least two cameras, wherein the initial first parameter set includes a plurality of first parameters corresponding one-to-one to a plurality of pixel points of the image; Fitting the initial first parameter set using a spline surface to obtain a fitting spline surface; The control points of the fitting spline surface are optimized based on a bundle adjustment algorithm to obtain a first parameter set of the at least two cameras. A wearable device, comprising: Binocular display module; Two lens barrels are arranged on the light-emitting side of the binocular display module, and at least one of the two lens barrels includes at least two cameras and an optical component arranged inside the lens barrel for collecting human eye images, and the at least two cameras are located between the binocular display module and the optical component. The wearable device according to claim 7, wherein the binocular display module includes a first display module and a second display module, and the two lens barrels include: A first lens barrel is arranged at a light-emitting side of the first display module, wherein a first camera and a second camera for collecting a first eye image of a human eye are arranged in the first lens barrel; The second lens barrel is arranged at the light-emitting side of the second display module, and a third camera and a fourth camera for collecting human eye images of the second eye are arranged in the second lens barrel. The wearable device according to claim 8, wherein the first camera and the second camera are symmetrically arranged in the first lens barrel relative to the axis of the first lens barrel; The third camera and the fourth camera are symmetrically arranged in the second lens barrel with respect to an axis of the second lens barrel. The wearable device according to claim 9, wherein the first camera and the second camera are both oriented toward the light-emitting side of the first display module and the orientations of the first camera and the second camera are aligned with the axis of the first lens barrel. The angles are equal; The third camera and the fourth camera are both facing the light-emitting side of the second display module, and the angles between the third camera and the fourth camera and the axis of the second lens barrel are equal. The wearable device according to claim 9, wherein the first camera and the second camera are both facing the first display module, a first reflection structure corresponding to the first camera and a second reflection structure corresponding to the second camera are arranged in the first lens barrel, the first reflection structure is used to reflect the light from the first eye to the first camera, and the second reflection structure is used to reflect the light from the first eye to the second camera; The third camera and the fourth camera are both facing the second display module. A third reflection structure corresponding to the third camera and a fourth reflection structure corresponding to the fourth camera are arranged in the second lens barrel. The third reflection structure is used to reflect the light from the second eye into the third camera, and the fourth reflection structure is used to reflect the light from the second eye into the fourth camera. A camera parameter calibration device, comprising: The receiving module is configured to: receive a plurality of images obtained by photographing a calibration reference object by at least two cameras; the wearable device further comprises a binocular display module, the lens barrel is arranged on a light-emitting side of the binocular display module, an optical component is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical component; A first determination module is configured to: determine a projection relationship between pixel points of the plurality of images and the calibration reference object; A second determination module is configured to: determine a first parameter set and a second parameter set of the at least two cameras according to the projection relationship; Among them, the first parameter set includes multiple first parameters, the first parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points, and the second parameter set includes at least one group of second parameters, and the at least one group of second parameters is used to indicate the posture relationship between the at least two cameras. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 6. A non-volatile computer-readable storage medium containing a computer program, when the computer program is executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 6. A computer program product comprises computer program instructions, and when the computer program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.