CN117392736A - Detection method and device for human eye gaze area and head-mounted augmented reality device - Google Patents

Abstract

This application discloses a method and device for detecting human eye gaze areas and a head-mounted augmented reality device. The method is applied to a head-mounted augmented reality device, wherein the head-mounted augmented reality device includes multiple light sources, and the shape of at least one light source is different from the shapes of other light sources. The method includes: acquiring a human eye image; determining the shape information of the bright spot based on the human eye image; determining the correspondence between the bright spot and the light source based on the shape information of the bright spot; based on the correspondence between the bright spot and the light source, Determine the area where the human eye is looking. This method determines the shape information of the bright spot. Even when the bright spot is missing in the human eye image, the head-mounted augmented reality device can quickly and accurately determine the correspondence between the bright spot and the light source based on the shape information of the light source. The correct correspondence between bright spots and light sources can ensure the accuracy of determining the human eye's gaze area, making the user's experience excellent.

Description

Detection method and device for human eye gaze area and head-mounted augmented reality device

Technical field

The present application relates to the field of augmented reality technology, and more specifically, to a method and device for detecting human eye gaze areas and a head-mounted augmented reality device.

Background technique

Eye tracking technology is widely used in head-mounted augmented reality devices, which can realize functions such as eye movement interaction and eye movement rendering.

The eye tracking technology provided by related technologies is as follows: the designated surface of the head-mounted augmented reality device is provided with an infrared light source. When the user wears the head-mounted display device, the infrared light source is in a glowing state. At this time, the near-eye camera in the head-mounted display device Collect the human eye image containing infrared bright spots, and then determine the corresponding relationship between the infrared light source and the infrared bright spots, so as to solve the user's human eye gaze area.

Among them, the scheme for determining the correspondence between infrared light sources and infrared bright spots is as follows: the head-mounted augmented reality device extracts the bright spots in the human eye image through the bright spot extraction algorithm, and then uses the brightest spot as the starting point, assuming multiple infrared light sources The mapping relationship with the infrared bright spots, and finally the probability of the assumed mapping relationship is solved through the Bayesian model, and the assumed mapping relationship with the highest probability is determined as the mapping relationship used to solve the human eye gaze area.

However, the above-mentioned scheme for determining the corresponding relationship between infrared light sources and infrared bright spots has the following defects: when the infrared light source irradiates the user's eye area, not all infrared light sources can be reflected in the corneal area of the glasses to form bright spots. Therefore, Missing bright spots will occur in the collected human eye images, that is, the number of bright spots in the human eye images is less than the number of infrared light sources. In this case, it is difficult for the Bayesian model to accurately determine the mapping relationship between the infrared light source and the infrared bright spot, resulting in the subsequent head-mounted augmented reality device being unable to determine the user's gaze area, which reduces the user experience.

Contents of the invention

Embodiments of the present application provide a method and device for detecting human eye gaze areas, and a head-mounted augmented reality device.

In the first aspect, some embodiments of the present application provide a method for detecting human eye gaze areas, which method is applied to head-mounted augmented reality devices. Wherein, the head-mounted augmented reality device includes multiple light sources, and the shape of at least one light source is different from the shapes of other light sources. The method includes: acquiring a human eye image, which is collected from the eye area of a user wearing a head-mounted display device, and the human eye image contains at least two bright spots; based on the human eye image, determining the shape information of the bright spots; Based on the shape information of the bright spot, the correspondence between the bright spot and the light source is determined; based on the correspondence between the bright spot and the light source, the human eye gaze area is determined.

In the second aspect, some embodiments of the present application also provide a device for detecting the human eye gaze area, which device is applied to a head-mounted augmented reality device. Wherein, the head-mounted augmented reality device includes multiple light sources, and the shape of at least one light source is different from the shapes of other light sources. The device includes: an image acquisition module, a first determination module, a second determination module and a third determination module. The image acquisition module is used to acquire a human eye image. The human eye image is collected from the eye area of a user wearing the head-mounted display device, and the human eye image contains at least two bright spots. The first determination module is used to determine the shape information of the bright spot based on the human eye image. The second determination module is used to determine the correspondence between the bright spot and the light source based on the shape information of the bright spot. The third determination module is used to determine the human eye gaze area based on the correspondence between the bright spot and the light source.

In a third aspect, some embodiments of the present application also provide a head-mounted augmented reality device. The head-mounted augmented reality device includes: multiple light sources, one or more processors, memory, and one or more applications. Wherein, the shape of at least one light source is different from the shape of other light sources, one or more application programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the above Detection method of human eye gaze area.

In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium that stores computer program instructions. Wherein, the computer program instructions can be called by the processor to execute the above-mentioned detection method of the human eye gaze area.

In a fifth aspect, embodiments of the present application further provide a computer program product, which, when executed, implements the above-mentioned detection method of the human eye gaze area.

This application provides a method and device for detecting the human eye gaze area and a head-mounted augmented reality device. The detection method in this application is applied to a head-mounted augmented reality device including multiple light sources. The shape of at least one light source among the multiple light sources is different from the shape of other light sources. Therefore, the multiple light sources illuminate the user's eye area to form a bright The shape of the spots is also different. The detection method in this application determines the shape information of the bright spot. Even when the bright spot is missing in the human eye image, the head-mounted augmented reality device can quickly and accurately determine the bright spot and the light source based on the shape information of the light source. The corresponding relationship between them, and then in the subsequent process, when determining the user's human eye gaze area, the correct correspondence between the bright spot and the light source can ensure the accuracy of determining the human eye gaze area.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 shows a schematic diagram of the implementation environment of a method for detecting human eye gaze areas provided by an embodiment of the present application.

Figure 2 shows a schematic diagram of a light source provided by an embodiment of the present application.

FIG. 3 shows a schematic flowchart of a method for detecting a human eye gaze area provided by the first embodiment of the present application.

FIG. 4 shows a schematic flowchart of a method for detecting a human eye gaze area provided by the second embodiment of the present application.

Figure 5 shows a schematic diagram of a human eye image and a bright spot mark image provided by an embodiment of the present application.

FIG. 6 shows a schematic flowchart of a method for detecting human eye gaze areas provided by the third embodiment of the present application.

Figure 7 shows a module block diagram of a device for detecting a human eye gaze area provided by an embodiment of the present application.

Figure 8 shows a module block diagram of a head-mounted augmented reality device provided by an embodiment of the present application.

Figure 9 shows a module block diagram of a computer-readable storage medium provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as limiting the present application.

In order to enable those in the technical field to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the scope of protection of this application.

This application provides a method and device for detecting the human eye gaze area and a head-mounted augmented reality device. The detection method in this application is applied to a head-mounted augmented reality device including multiple light sources. The shape of at least one light source among the multiple light sources is different from the shape of other light sources. Therefore, the multiple light sources illuminate the user's eye area to form a bright The shape of the spots is also different. The detection method in this application determines the shape information of the bright spot. Even when the bright spot is missing in the human eye image, the head-mounted augmented reality device can quickly and accurately determine the bright spot and the light source based on the shape information of the light source. The corresponding relationship between them, and then in the subsequent process, when determining the user's human eye gaze area, the correct correspondence between the bright spot and the light source can ensure the accuracy of determining the human eye gaze area.

In order to facilitate a detailed description of the solution of the present application, the implementation environment in the embodiment of the present application will be introduced below with reference to the accompanying drawings. Please refer to Figure 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a head-mounted augmented reality device 100. The head-mounted augmented reality device 100 may be augmented reality glasses, an augmented reality helmet, or the like.

The head-mounted augmented reality device 100 refers to a device that can provide augmented reality (Augmented Reality, AR) technology. AR technology is a new technology that integrates real world information and virtual world information. Users can travel in the virtual world based on the head-mounted augmented reality device 100 Entertainment and learning in an environment that is integrated with the real world. In this embodiment of the present application, the head-mounted augmented reality device 100 can implement eye movement interaction, that is, the head-mounted augmented reality device 100 can render virtual images in the area where the user's eyes are looking.

In the embodiment of the present application, the head-mounted augmented reality device 100 includes a body 10 , a plurality of light sources 30 and an image collection device 50 . The plurality of light sources 30 are evenly distributed on the designated surface of the body 10. The "designated surface" may be the surface of the body 10 facing the user when the head-mounted augmented reality device 100 is in the wearing state. In the embodiment of the present application, the shape of at least one light source 30 among the plurality of light sources 30 is different from the shape of other light sources 30 , where the “shape of the light source 30 ” should be understood as the shape of the light emitting area on the light source 30 . Specifically, the shape of the light source 30 can be a highly distinctive shape such as a circle, a rectangle, a cross, etc., so that the bright spot corresponding to the light source 30 can be affected by smaller distortion caused by the image acquisition device 50, making the head-mounted The augmented reality device 100 can subsequently successfully identify the shape information of bright spots in the human eye image.

Please refer to FIG. 2 , which shows a schematic diagram of a light source 30 . In FIG. 2 , the body 10 includes a first mirror frame 101 and a second mirror frame 103 . Each frame is provided with six light sources 30 , and two of the six light sources 30 have the same shape. It should be noted here that each light source 30 corresponds to a unique number, and the number can be preset by relevant personnel. For example, the two circular light sources are light source No. 1 and light source No. 4, the two longitudinal rectangular light sources are light source No. 2 and light source 3, and the two horizontal rectangular light sources are light source No. 5 and light source 6. , the embodiment of the present application does not specifically limit the shape, number, quantity, etc. of the light sources 30 .

The image capture device 50 is also disposed on a designated surface of the body 10 for capturing images of the user's eyes. Specifically, when the above-mentioned light source 30 is in a light-emitting state, the user's eye area will form a bright spot corresponding to the light source 30. Since the shapes of the multiple light sources 30 in this application are different, the shape of the bright spot formed when the light source 30 is illuminated in the user's eye area is Not the same either. Therefore, after the user's human eye image is collected by the image acquisition device 50, the shape information of the bright spot in the human eye image is then determined. Even in the case where the bright spot is missing, the light source bright spot can be accurately determined based on the shape information of the light source 30. The above correspondence relationship can be used to subsequently determine the user's human eye gaze area.

In the embodiment of the present application, the type of the image acquisition device 50 corresponds to the type of the light source 30 . In some embodiments, the light source 30 may be an infrared light source, and the image acquisition device 50 is an infrared camera. In other embodiments, the light source 30 may be a visible light source, and the image acquisition device 50 is a digital camera. In some embodiments, the number of image acquisition devices 50 may be two, which are respectively provided in the first frame 101 and the second frame 103 and are used to collect the left eye image and the right eye image of the user respectively. In the embodiment of the present application, the image The type, model, quantity, etc. of the collection device 50 are not specifically limited.

In the embodiment of the present application, after the head-mounted augmented reality device 100 collects the human eye image, it determines the shape information of the bright spots in the human eye image based on the pre-trained classification model, and then based on the shape information of the light source 30, through Bayesian The model can accurately determine the corresponding relationship between the bright spot and the light source 30 .

The above-mentioned pre-trained classification model and Bayesian model can be stored in the head-mounted augmented reality device 100 to increase the calculation speed and thereby improve the efficiency of determining the correspondence between light source bright spots. The above two models can also be set in the server, and the head-mounted augmented reality device 100 interacts with the server online while in working condition to obtain the output results of the pre-trained classification model and Bayesian model. In the above manner, the The calculation amount of the head-mounted augmented reality device 100 saves the power consumption of the head-mounted augmented reality device 100 . In the embodiment of the present application, only the pre-trained classification model and the Bayesian model are installed in the head-mounted augmented reality device 100 as an example for description.

In some embodiments, the head-mounted augmented reality device 100 further includes a rendering module and a sensor component. The above rendering module is used to render images. In some embodiments, the rendering module is used to render images within the gaze area. The sensor component is used to monitor motion information, status information, etc. of the head-mounted augmented reality device 100. The sensor component may include an acceleration sensor, a distance sensor, a pressure sensor, etc., which are not limited in the embodiments of the present application.

Please refer to Figure 3. Figure 3 schematically illustrates a method for detecting human eye gaze areas provided by the first embodiment of the present application. This method is applied to the head-mounted augmented reality device in Figure 1, wherein the head-mounted augmented reality device Realistic devices include multiple light sources, at least one of which has a different shape than the other light sources. Specifically, the method includes steps S310 to S340.

Step S310: Obtain human eye image.

The human eye image is collected from the eye area of the user wearing the head-mounted display device, and the human eye image contains at least two bright spots. When multiple light sources are in the emitting state, the user's eye area is irradiated to form bright spots corresponding to the light sources on the eye. At this time, the human eye image obtained by wearing the head-mounted display device to collect images of the user's eye area also includes The aforementioned bright spots.

In some embodiments, when the head-mounted augmented reality device detects that the wearing of the head-mounted augmented reality device is completed, the above-mentioned human eye image is collected through the image acquisition device. Optionally, a wearing detection device is provided on the head-mounted augmented reality device, and the wearing detection device detects whether the head-mounted augmented reality device is in a wearing state.

The above-mentioned wearing detection device may be a pressure sensor, which is arranged at a position where the head-mounted augmented reality device contacts the human body when the user wears the head-mounted augmented reality device. Taking the head-mounted augmented reality device as augmented reality glasses as an example, the above pressure sensor is arranged at a position where the augmented reality glasses are in contact with the bridge of the nose, or can also be arranged at the end of the temple of the augmented reality glasses. When the pressure value detected by the pressure sensor is greater than the preset pressure value, it means that the user has completed wearing the head-mounted augmented reality device.

The above-mentioned wearing detection device can also be a distance sensor. Since the distance between the user and each point on the head-mounted augmented reality device remains unchanged after wearing the head-mounted augmented reality device, the distance sensor can be used to detect whether the user has completed the head-mounted augmentation. When wearing a reality device, take the head-mounted augmented reality device as augmented reality glasses as an example. The above distance sensor can be set on a designated surface (that is, under the premise that the user wears the head-mounted augmented reality device, the head-mounted augmented reality device faces the user. side surface), when the distance value detected by the distance sensor falls within the preset distance range, it means that the user has completed wearing the head-mounted augmented reality device.

In other embodiments, the head-mounted augmented reality device collects human eye images through the image acquisition device after receiving the work instruction. Optionally, a start button is provided on the head-mounted augmented reality device. When the head-mounted augmented reality device receives a pressing operation signal for the start button, it receives the work instruction. Optionally, the display interface of the external device used to control the head-mounted augmented reality device includes working controls of the head-mounted augmented reality device. After receiving the trigger signal for the above-mentioned working controls, the external device sends the working controls to the head-mounted augmented reality device. instruction. The above-mentioned external device may be a terminal device such as a smartphone or a tablet computer.

It should be noted that in order to ensure that the collected human eye images include bright spots corresponding to multiple light sources on the designated surface of the head-mounted augmented reality device, the above-mentioned multiple light sources need to be in a light-emitting state. Before displaying the eye image, it is first detected whether the plurality of light sources are in the light-emitting state. If not, the plurality of light sources are controlled to switch to the light-emitting state. Optionally, after acquiring the human eye image, the head-mounted augmented reality device controls the above-mentioned multiple light sources to switch to an extinguished state to avoid discomfort to the human eyes caused by the light source being permanently on.

Step S320: Determine the shape information of the bright spot based on the human eye image.

Since there are differences in the shapes of the light sources in this application, there are also differences in the shapes of the bright spots formed in the user's eye area. For example, a circular light source corresponds to a circular bright spot, and a rectangular light source corresponds to a rectangular bright spot. Therefore, in this application, by determining the shape information of the bright spots in the human eye image, a basis is provided for subsequent determination of the correspondence between the bright spots and the light source. Specifically, the specific implementation manner of determining the shape information of the bright spot will be described in detail in the embodiments below.

Step S330: Based on the shape information of the bright spot, determine the corresponding relationship between the bright spot and the light source.

The bright spot corresponds to the light source, which represents the bright spot formed by the reflection of the human eye when the light source is in the emitting state. In some embodiments, there is only one light source of a certain shape (for example, a circular light source). When it is determined that the shape information of a certain bright spot is circular, it can be determined that the circular bright spot is the same as the circular light source. Correspondence between shaped light sources.

In other embodiments, there are multiple light sources of a certain shape (for example, circular light sources). Please refer to Figure 2 again. There are two circular light sources in Figure 2, namely light source No. 1 and light source No. 4. light source, when it is determined that the shape information of a certain bright spot is circular, multiple possible mapping relationships between the circular bright spot and the circular light source can be determined, that is, the circular bright spot may Corresponds to circular light source No. 1, and may also correspond to circular light source No. 4.

In order to further determine the unique corresponding relationship between the bright spot and the light source from multiple possible mapping relationships. In this application, the human eye image contains at least two bright spots, and the head-mounted augmented reality device determines the correspondence between the bright spots and the light source based on the shape information and position information of the at least two bright spots. The position information of the bright spot represents the position of the bright spot in the human eye image, that is, the pixel coordinates corresponding to the bright spot. Taking the coordinate value corresponding to the pixel coordinate as a plane rectangular coordinate system as an example, the pixel coordinates corresponding to the bright spot can be recorded is (x, y). Taking the human eye image as a right eye image as an example, the right eye image contains two bright spots, namely circular bright spot No. 1 and circular bright spot No. 2. Among them, the pixel coordinates of the circular bright spot No. 1 are (x1, y1), and the pixel coordinates of the circular bright spot No. 2 are (x2, y2). The head-mounted augmented reality device can determine the bright spots based on the position information of the two bright spots. The relative positional relationship between spots, for example, when the abscissa value x1 is less than the abscissa value x2, it can be determined that the circular bright spot No. 1 is to the left of the circular bright spot No. 2. Specifically, the specific implementation manner of determining the position information of the bright spot will be described in detail in the embodiments below.

In some embodiments, the head-mounted augmented reality device determines the corresponding relationship between the bright spots and the light source based on the relative positional relationship between the bright spots and a preset Bayesian model. The Bayesian model is a probabilistic model that stores the probabilities of multiple events in advance. In this embodiment, the "probability of event occurrence" refers to the probability of a bright spot corresponding to a light source. Researchers obtain multiple right-eye images in advance and label each bright spot in the right-eye image with the corresponding light source number. Based on the labeling results, the probability of the bright spot corresponding to the light source can be obtained. For example, the probability that the circular bright spot located on the left side of the image corresponds to the circular light source No. 1 is 95%, and the probability that it corresponds to the circular light source No. 4 is 5%. The above probability can be stored in the Bayesian model in the form of a probability matrix. When the head-mounted augmented reality device determines that there are two circular bright spots in the right eye image, it can be determined by looking up the above probability matrix: located on the left side of the image The No. 1 circular bright spot corresponds to the No. 1 circular light source, and the No. 2 circular bright spot corresponds to the No. 4 circular light source.

In other embodiments, the head-mounted augmented reality device determines the corresponding relationship between the bright spots and the light source based on the relative positional relationship between the bright spots and the position information of the light source. The position information of the light source represents the position of multiple light sources on the body. For example, in Figure 2, compared with the circular light source No. 4, the circular light source No. 1 is closer to the center of the body (that is, in the head-mounted augmented reality device When in the wearing state, the circular light source No. 1 is closer to the bridge of the user's nose). Therefore, in the right eye image, the bright spot corresponding to the circular light source No. 1 is located to the left of the bright spot corresponding to the circular light source No. 4, also That is, based on the above position information, the head-mounted augmented reality device can determine that the circular bright spot No. 1 corresponds to the circular light source No. 1, and the circular bright spot No. 2 corresponds to the circular light source No. 4.

This application can limit the search range of light sources by setting up multiple light sources with different shapes. That is, the head-mounted augmented reality device only needs to search for light sources with the same shape as the bright spots based on the shape information of the bright spots, thereby speeding up the process of bright spots and bright spots. The speed of determination of correspondence between light sources.

Step S340: Determine the human eye's gaze area based on the correspondence between the bright spot and the light source.

The specific implementation of the head-mounted augmented reality device to determine the human eye gaze area will be described in detail in the embodiments below.

This application provides a method for detecting human eye gaze areas. The detection method in this application is applied to a head-mounted augmented reality device including multiple light sources. The shape of at least one light source among the multiple light sources is different from the shape of other light sources. Therefore, the multiple light sources illuminate the user's eye area to form a bright The shape of the spots is also different. The detection method in this application determines the shape information of the bright spot. Even when the bright spot is missing in the human eye image, the head-mounted augmented reality device can quickly and accurately determine the bright spot and the light source based on the shape information of the light source. The corresponding relationship between them, and then in the subsequent process, when determining the user's human eye gaze area, the correct correspondence between the bright spot and the light source can ensure the accuracy of determining the human eye gaze area.

Please refer to Figure 4. Figure 4 schematically shows a method for detecting the human eye gaze area provided by the second embodiment of the present application. In this method, the determination process of the position information and shape information of the bright spot is introduced in detail. , specifically, this method may include step S410 to step S460.

Step S410: Obtain human eye image.

For the implementation of step S410, reference may be made to the detailed introduction in step S310, which will not be described again here.

Step S420: Determine the location information of the bright spot based on the human eye image.

The position information of the bright spot represents the position of the bright spot in the human eye image, that is, the corresponding pixel coordinates of the bright spot in the human eye image. Since the bright spot area is an area formed by the light source, the pixel value of the bright spot area will be greater than the pixel value of the non-bright spot area.

In some embodiments, a binarization algorithm is provided in the head-mounted augmented reality device, and the binarization algorithm can separate the bright spot areas and non-bright spot areas in the human eye image. Specifically, a first specified threshold is set in the binarization algorithm. The binarization algorithm sets pixel values greater than or equal to the first specified threshold to 1, and sets pixel values smaller than the first specified threshold to 0. Multiple The pixel area with a pixel value of 1 is the bright spot area. Wherein, the first specified threshold is determined based on the pixel value of the bright spot area. After determining the bright spot area, the head-mounted augmented reality device then obtains the pixel coordinates corresponding to the bright spot area, that is, the location information of the bright spot.

In other embodiments, the position information of the bright spot is determined by performing a convolution operation on the human eye image. Since there are at least two bright spots in the human eye image, the location information of the bright spot is determined through at least two rounds of determination processes in this application. That is, in each round of determination process, the position information of a bright spot is determined. location information. Specifically, the i-th round of determination process among at least two rounds of determination processes includes step A100 to step A200, and i is a positive integer.

Step A100: Perform a convolution operation on the specified image based on a preset convolution kernel to obtain an intermediate image.

In the embodiment of this application, the size of the preset convolution kernel is set by default according to the size of the bright spot in the human eye image. The researchers take multiple human eye images in advance and determine the size of the bright spot. For example, if the size of the bright spot is 9*8, 8*10, etc., then the size of the preset convolution kernel can be 9*9, 10*10, etc., and the shape of the convolution kernel can be round, Square, and other shapes, this application does not limit the size and shape of the convolution kernel.

The head-mounted augmented reality device performs a convolution operation on the specified image by obtaining the pre-stored convolution kernel to obtain an intermediate image.

The following explains the specified image.

When i is 1, the specified image is a human eye image. That is, in the first round of determination process, the head-mounted augmented reality device performs a convolution operation on the human eye image based on the preset convolution kernel to obtain an intermediate image.

When i is greater than 1, the specified image is obtained in the following way: set the pixel value of the target area in the human eye image to the specified pixel value to obtain the specified image. The target area is determined based on the position information of the first i-1 bright spots. .

When i is greater than 1, in the i-th round of determination process, it is necessary to eliminate the influence of the first i-1 bright spots on the determination of the position information of the i-th bright spot. Therefore, it is necessary to eliminate the influence of the first i-1 bright spots in the human eye image. The area where -1 bright spot is located (that is, the target area) is processed, that is, the pixel value of the target area is set to the specified pixel value.

After determining the sub-region where each of the first i-1 bright spots is located, the head-mounted augmented reality device determines the union area of the sub-regions as the target area. Specifically, the sub-region corresponding to each of the first i-1 bright spots can be a rectangular area, a circular area, or an area of other shapes, and the size of the sub-region is greater than or equal to the size of the bright spot, and the The position of the center pixel of the sub-region is the position corresponding to the bright spot, that is, the sub-region can cover the bright spot. Taking i as 3 as an example, since the position information of the first bright spot and the second bright spot has been determined, in the third round of determination process, the corresponding positions of the first bright spot and the second bright spot are determined respectively. sub-region, wherein the sub-region corresponding to the i-th bright spot is determined based on the position information of the i-th bright spot, i is equal to 1 or 2, and then the union area of the sub-regions is determined as the target area.

When the head-mounted augmented reality device determines the target area, it further sets the pixel value of the target area to the specified pixel value. For example, the specified pixel value is 0, that is, the image corresponding to the target area is set to black. Since the size of each sub-region in the target area is greater than or equal to the size of the bright spots, the images corresponding to the first i-1 bright spots are also set to black. Through the above processing, the first i-1 bright spot pairs can be eliminated. The influence of the i-th bright spot when the position information is determined makes the position information of the i-th bright spot more accurate.

Step A200: determine the position information of the target pixel point in the intermediate image as the position information of the i-th bright spot.

The pixel value of the target pixel is the maximum value of all pixel values in the intermediate image. Specifically, the head-mounted augmented reality device sorts all pixel values in the intermediate image, determines the maximum value, and determines the pixel corresponding to the maximum value as the target pixel. Sorting processing can be completed through sorting algorithms such as bubble sorting and selection sorting, and is not limited here. In some embodiments, the number of pixels corresponding to the maximum value is multiple, then any one pixel among the multiple pixels is determined as the target pixel.

In some embodiments, in order to ensure that the position of the target pixel corresponds to the position of the i-th bright spot, when the head-mounted augmented reality device determines the target pixel, it also needs to further determine the pixel value corresponding to the target pixel. Whether it is greater than or equal to the specified pixel value. The specified pixel value is set by the researcher by default. If the pixel value corresponding to the target pixel is greater than or equal to the specified pixel value, the position information of the target pixel point is determined as the position information of the i-th bright spot. . If the pixel value corresponding to the target pixel point is less than the specified pixel value, step A200 is no longer performed, and step S430 is performed.

By setting specified pixel values, embodiments of the present application can avoid continuing to determine the position information of bright spots when there are no bright spots in the specified image, ensuring the accuracy of the position information of bright spots.

Step S430: Mark the bright spots in the human eye image based on the position information of the bright spots to obtain a bright spot marked image.

A bright spot marked image refers to an image that includes only one bright spot. That is, the number of bright spot mark images is the same as the number of bright spots. As an implementation manner, the head-mounted augmented reality device determines a sub-image of the human eye image based on the position information of the bright spot, and then determines the sub-image as a bright spot mark image. Please refer to Figure 5. Figure 5 schematically shows a schematic diagram of a human eye image 510 and a bright spot mark image 520 provided by an embodiment of the present application. In FIG. 5 , the bright spots 521 in the bright spot mark image 520 are bright spots corresponding to the vertical rectangular light source. Specifically, the image size of the sub-image is set by default by the developer and is larger than the size of the bright spot. The central pixel position information of the sub-image is determined by the position information of the bright spot, that is, the bright spot is located in the sub-image (that is, A bright spot marks the center of the image).

As another implementation manner, the head-mounted augmented reality device determines a sub-image of the human eye image based on the position information of the bright spot, and sets other areas in the human eye image other than the sub-image to specified pixel values, for example, specified pixels The value is 0 (that is, only the bright spots in the sub-image are retained, and other image areas outside the sub-image are set to black). Finally, the processed human eye image is determined as the bright spot mark image.

Step S440: Determine the shape information of the bright spot based on the bright spot mark image.

In this embodiment of the present application, the shape information of the bright spot is determined by the classification model. The head-mounted augmented reality device inputs the bright spot mark image into the pre-trained classification model, and the output of the classification model is the shape information of the bright spot. Specifically, step S440 may include step S4410.

Step S4410: Predict the bright spot mark image through a classification model to obtain the shape information of the bright spot in the bright spot mark image.

The classification model is obtained by training the initial model based on at least one training sample. At least one training sample includes a bright spot image, and the bright spot image is annotated with the true shape information of the bright spot. It should be noted here that the classification model provided in this application is a classification model based on a neural network. Through the classification model of the neural network, the bright spot mark image is predicted and processed, and the shape information of the bright spot can be accurately obtained. The specific model structure and training method of the neural network classification model are introduced in the following embodiments.

In other embodiments, the classification model can also be other classification prediction models, for example, a model based on the K-nearest Neighbors (KNN) algorithm, a logistic regression model, or a model based on Linear Discriminant Analysis (LDA). Models, models based on Quadratic Discriminant Analysis (QDA), etc. are not specifically limited in this application.

Step S450: Determine the corresponding relationship between the bright spot and the light source based on the shape information of the bright spot.

For the implementation of step S450, reference can be made to the detailed introduction in step S330, which will not be described again here.

Step S460: Determine the human eye's gaze area based on the correspondence between the bright spot and the light source.

This application provides a method for detecting human eye gaze areas. In this embodiment, the method of determining the position information and shape information of the bright spot is introduced in detail, which provides a basis for determining the subsequent correspondence between the bright spot and the light source, so that the corresponding relationship is determined based on the shape information of the bright spot. More accurate and reliable.

Please refer to Figure 6. Figure 6 schematically shows a method for detecting the human eye gaze area provided by the third embodiment of the present application. In this method, the determination process of the human eye gaze area is introduced in detail. Specifically, The method may include steps S610 to S680.

Step S610: Obtain human eye image.

Step S620: Determine the shape information of the bright spot based on the human eye image.

Step S630: Determine the corresponding relationship between the bright spot and the light source based on the shape information of the bright spot.

Step S640: Determine the spatial position information of the cornea center based on the correspondence between the bright spot and the light source.

As an implementation manner, when the bright spot establishes a corresponding relationship with the light source, the head-mounted augmented reality device determines the position information of the camera's optical center by reading the hardware information of the image acquisition device, and then based on the bright spot, the corresponding light source, and the cornea Based on the principle that the center and the four points of the camera's optical center are coplanar, the spatial position information of the cornea center is obtained through spatial geometric relationships.

What needs to be explained here is that the spatial position information of the cornea center represents the position information of the cornea center in the target space, and the target space is the three-dimensional space where the head-mounted augmented reality device is located. Specifically, the above three-dimensional space can be described based on a preset three-dimensional space rectangular coordinate system, and the spatial position information of the cornea center can be the corresponding coordinates in the corresponding coordinate system.

Step S650: Based on the human eye image, determine the position information of the pupil center and the pupil outline information in the human eye image.

Because the pupil area of the human eye is darker in color and approximately round in shape. Therefore, in this application, the pupil area is determined through a binarization operation, and then a curve is used to fit the pupil area to determine the pupil outline information and the position information of the pupil center. Specifically, step S650 may include steps S6510 to S6520.

Step S6510, determine the area in the human eye image whose pixel value is less than or equal to the preset pixel threshold as the pupil area.

The preset pixels are set by default by the head-mounted augmented reality device, and can also be adjusted by developers based on actual processing results. The head-mounted augmented reality device determines the area in the human eye image with a pixel value less than or equal to the preset pixel threshold as the pupil area, and sets the pixel values of all pixels in the pupil area to 0 (that is, the image is set to black). In addition, the head-mounted augmented reality device determines the area in the human eye image with a pixel value greater than the preset pixel threshold as a non-pupil area, and sets the pixel values of all pixels in the non-pupil area to 1 (that is, the image is set to white ). The head-mounted augmented reality device realizes the segmentation of the pupil area and non-pupil area through the above binary operation.

Step S6520: Based on the pupil area, determine the position information of the pupil center and the outline information of the pupil.

In this embodiment of the present application, the head-mounted augmented reality device uses a preset fitting curve to fit the pupil area to obtain the outline information of the pupil, and then determines the position information of the pupil center based on the fitted curve. Specifically, the preset fitting curve is an elliptic curve, and the head-mounted augmented reality device fits the elliptic curve through the least squares method to obtain the fitted elliptic curve, that is, the contour information of the pupil. Then, based on the fitted elliptic curve, the midpoint coordinates of the elliptic curve are determined as the position information of the pupil center.

Step S660: Determine the spatial position information of the pupil center based on the spatial position information of the cornea center, the position information of the pupil center, and the pupil outline information.

In the embodiment of this application, the outline information of the pupil is described by an elliptic curve. The head-mounted augmented reality device can determine the long axis length and the short axis length of the ellipse based on the pupil outline information, and then based on the spatial position information of the corneal center and The position information of the pupil center can be determined based on the pinhole imaging principle and the law of refraction. Similarly, the spatial position information of the pupil center is also described using the same coordinate system as the spatial position information of the cornea center. That is, the spatial position information of the pupil center and the spatial position information of the cornea center are coordinates in the same coordinate system.

Step S670: Determine the visual axis based on the spatial position information of the cornea center and the spatial position information of the pupil center.

The visual axis is used to represent the gaze direction of the human eye. When the spatial position information of the cornea center and the pupil center are known, the head-mounted augmented reality device determines the optical axis based on the spatial geometric relationship. That is, the straight line where the optical axis is located is determined based on the coordinates corresponding to the center of the cornea and the center of the pupil. The head-mounted augmented reality device then determines the angle information between the optical axis and the visual axis based on the hardware information of the image acquisition device, and then adjusts the angle of the straight line where the optical axis is based on the angle information. The adjusted straight line is the visual axis. The straight line where it is located.

Step S680: Determine the human eye gaze area based on the visual axis.

The head-mounted augmented reality device obtains the visual axes corresponding to the left eye and the right eye respectively, and then determines the intersection point of the two visual axes (that is, the visual gaze point). Then, the area where the visual fixation point is located is determined as the human eye fixation area. Among them, the size and shape of the human eye gaze area are set by default by the head-mounted augmented reality device. For example, the human eye gaze area is a spherical area with the visual gaze point as the center and a radius of a specified radius.

This application provides a method for detecting human eye gaze areas. In this embodiment, the method for determining the human eye gaze area is introduced in detail. Based on this method, the head-mounted augmented reality device can accurately determine the human eye gaze area based on the correspondence between the bright spot and the light source, which improves the user's use. experience.

The training process of the classification model is explained below. The classification model can be trained by a server or by a head-mounted augmented reality device, which is not limited in the embodiments of this application. The training process of the classification model includes the following steps: iteratively train the initial model using the sample data set to obtain the classification model.

The sample dataset includes bright spot images annotated with real shape information. The annotation process of bright spot images is done manually. The number of bright spot images is actually set according to the accuracy requirements of the classification model. The higher the accuracy requirements of the classification model, the greater the number of bright spot images.

The initial model can include input layer, hidden layer and output layer. The embodiments of the present application do not limit the structure of the hidden layer, which can be selected by skilled personnel based on experience. For example, the hidden layer can include multiple convolutional layers.

Among them, the specific implementation methods of each model training include: inputting the bright spot image into the initial model for prediction processing to obtain the predicted shape information of the bright spot image; based on the relative error and loss function between the predicted shape information and the annotated real shape information , adjust the parameters of the initial model, and use the adjusted initial model as the initial model for the next model training.

Optionally, if the server detects that the iteration stop condition is met during the iterative training process, it generates a classification model. The stop iteration condition may be that the number of parameter adjustments is greater than the preset number, or the relative error between the annotated real shape information and the predicted shape information of the bright spot image is less than the preset error. Both the preset times and the preset error can be actually set according to the accuracy requirements of the classification model. The higher the accuracy requirements of the classification model, the greater the preset times and the smaller the preset error.

Loss functions include but are not limited to: absolute value loss function, 0-1 loss function, logarithmic loss function, square loss function, perceptual loss function, cross-entropy loss function, etc.

Optionally, the server uses a preset algorithm to adjust the parameters of the initial model (such as various parameters in the hidden layer) based on the relative error and loss function between the annotated real shape information and the predicted shape information. The above-mentioned preset algorithm can be a gradient descent algorithm, including a batch gradient descent (Batch Gradient Descent, BGD) algorithm, a stochastic gradient descent (Stochastic Gradient Descent, SGD) algorithm, a mini-batch gradient descent (Mini-Batch Gradient Descent, MBGD) algorithm, etc. .

It should be noted that the server can also test the trained classification model based on the test bright spot image. The test bright spot image is also annotated with real shape information. The server inputs the test bright spot image into the classification model, and compares the predicted shape information output by the classification model with the annotated real shape information. If the relative error between the two is less than the predicted Assuming an error, it means that the parameters of the classification model do not need to be adjusted further.

Embodiments of the present application provide a method for training a classification model. The classification model trained based on this method can accurately predict the shape of bright spots in human eye images.

Please refer to FIG. 7 , which schematically illustrates a human eye gaze area detection device 700 provided by an embodiment of the present application. The device 700 is applied to a head-mounted augmented reality device. Wherein, the head-mounted augmented reality device includes multiple light sources, and the shape of at least one light source is different from the shapes of other light sources. The device 700 includes: an image acquisition module 710, a first determination module 720, a second determination module 730 and a third determination module 740. The image acquisition module 710 is used to acquire a human eye image. The human eye image is collected from the eye area of the user wearing the head-mounted display device, and the human eye image contains at least two bright spots. The first determination module 720 is used to determine the shape information of the bright spot based on the human eye image. The second determination module 730 is configured to determine the corresponding relationship between the bright spot and the light source based on the shape information of the bright spot. The third determination module 740 is used to determine the human eye gaze area based on the correspondence between the bright spot and the light source.

The detection device in this application determines the shape information of the bright spot. Even when the bright spot is missing in the human eye image, the head-mounted augmented reality device can quickly and accurately determine the bright spot and the light source based on the shape information of the light source. The corresponding relationship between them, and then in the subsequent process, when determining the user's human eye gaze area, the correct correspondence between the bright spot and the light source can ensure the accuracy of the human eye gaze area determination, making the user's use experience excellent.

In some embodiments, for each bright spot, the first determination module 720 is also used to determine the position information of the bright spot based on the human eye image. The position information of the bright spot represents the position of the bright spot in the human eye image; based on the bright spot The spot position information marks the bright spot in the human eye image to obtain a bright spot mark image; based on the bright spot mark image, the shape information of the bright spot is determined.

In some embodiments, the first determination module 720 is also configured to determine the location information of the bright spot through at least two rounds of determination processes. Specifically, the first determination module 720 includes a first determination sub-module (not shown in the figure) and a second determination sub-module (not shown in the figure). In the i-th round of determination process among at least two rounds of determination processes, i is a positive integer, and the first determination sub-module is used to perform a convolution operation on the specified image based on a preset convolution kernel to obtain an intermediate image. The second determination sub-module is used to determine the position information of the target pixel point in the intermediate image as the position information of the i-th bright spot, and the pixel value of the target pixel point is the maximum value of all pixel values in the intermediate image. Among them, when i is 1, the specified image is a human eye image; when i is greater than 1, the specified image is obtained in the following way: set the pixel value of the target area in the human eye image to the specified pixel value, and obtain the specified Image, the target area is determined based on the position information of the first i-1 bright spots.

In some embodiments, the first determination module 720 is also used to perform prediction processing on the bright spot mark image through a classification model to obtain the shape information of the bright spots in the bright spot mark image. The classification model is based on at least one training sample and performs prediction on the initial model. After training, at least one training sample includes a bright spot image, and the bright spot image is annotated with the true shape information of the bright spot.

In some embodiments, the third determination module 740 is also used to determine the spatial position information of the cornea center based on the correspondence between the bright spot and the light source. The spatial position information of the cornea center represents the position information of the cornea center in the target space, The target space is the three-dimensional space where the head-mounted augmented reality device is located; based on the human eye image, the position information of the pupil center and the pupil outline information in the human eye image are determined; based on the spatial position information of the cornea center, the position information of the pupil center and the pupil's contour information Contour information determines the spatial position information of the pupil center; determines the visual axis based on the spatial position information of the cornea center and the pupil center, and the visual axis is used to characterize the gaze direction of the human eye; determines the human eye gaze area based on the visual axis .

In some embodiments, the third determination module 740 is also used to determine the area in the human eye image with a pixel value less than or equal to the preset pixel threshold as the pupil area; based on the pupil area, determine the position information of the pupil center and the outline information of the pupil. .

In some embodiments, the device 700 also includes a training module (not shown in the figure). The training module is used to train the initial model using a sample data set to obtain a classification model. The sample data set includes bright spots marked with real shape information. image; the training module is specifically used to input the bright spot image into the initial model for prediction processing to obtain the predicted special shape information of the bright spot image; based on the relative error and loss function between the predicted shape information and the annotated real shape information, the initial The parameters of the model are adjusted, and the initial model after parameter adjustment is used as the initial model for the next model training.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules.

Please refer to Figure 8, which shows that an embodiment of the present application also provides a head-mounted augmented reality device 800. The head-mounted augmented reality device 800 includes: multiple light sources 810, one or more processors 820, a memory 830 and a or multiple applications. Wherein, the shape of at least one light source 810 is different from the shape of other light sources 810, one or more application programs are stored in the memory 830 and configured to be executed by one or more processors 820, and the one or more application programs are configured Used to perform the methods described in the above embodiments.

Processor 820 may include one or more processing cores. The processor 820 uses various interfaces and lines to connect various parts within the entire battery management system, by running or executing instructions, programs, code sets or instruction sets stored in the memory 830, and calling data stored in the memory 830, Performs various functions of the battery management system and processes data. Optionally, the processor 820 may adopt at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). implemented in hardware form. The processor 820 may integrate one or a combination of a central processing unit 820 (Central Processing Unit, CPU), a graphics processor 820 (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the display content; and the modem is used to handle wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 820 and may be implemented solely through a communication chip.

The memory 830 may include a random access memory 830 (Random Access Memory, RAM) or a read-only memory 830 (Read-Only Memory). Memory 830 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 830 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing various method embodiments described below, etc. The storage data area can also store data created during use of the electronic device (such as phone book, audio and video data, chat record data), etc.

Please refer to Figure 9, which shows that an embodiment of the present application also provides a computer-readable storage medium 900. The computer-readable storage medium 900 stores computer program instructions 910. The computer program instructions 910 can be called by the processor for execution. The method described in the above embodiments.

The computer-readable storage medium may be electronic memory such as flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes non-transitory computer-readable storage medium (non-transitory computer-readable storage medium). The computer-readable storage medium 900 has storage space for computer program instructions 910 that perform any method steps in the above-described methods. These computer program instructions 910 may be read from or written into one or more computer program products.

The above are only preferred embodiments of the present application and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, they are not intended to limit the present application. Any person skilled in the art may Without departing from the scope of the technical solution of the present application, the technical content disclosed above can be used to make some changes or modifications to equivalent embodiments with equivalent changes. Any brief modifications, equivalent changes and modifications made to the above embodiments still fall within the scope of the technical solution of the present application.

Claims (10)

1. A method of detecting an eye gaze area, the method being applied to a head-mounted augmented reality device comprising a plurality of light sources, wherein at least one of the light sources is shaped differently from the other light sources, the method comprising:

acquiring a human eye image, wherein the human eye image is acquired from an eye area of a user wearing the head-mounted display device, and the human eye image comprises at least two bright spots;

determining shape information of the bright spots based on the human eye image;

Based on the shape information of the bright spots, determining the corresponding relation between the bright spots and the light source;

and determining a human eye gazing area based on the corresponding relation between the bright spots and the light source.

2. The method of claim 1, wherein for each of the bright spots, the determining shape information of the bright spot based on the human eye image comprises:

determining position information of the bright spots based on the human eye image, wherein the position information of the bright spots represents the positions of the bright spots in the human eye image;

marking the bright spots in the human eye image based on the position information of the bright spots to obtain a bright spot marking image;

and determining shape information of the bright spots based on the bright spot mark image.

3. The method of claim 2, wherein the determining location information of the bright spots based on the human eye image comprises:

determining the position information of the bright spots through at least two rounds of determining processes; the ith round of determining process in the at least two rounds of determining process comprises the following steps that i is a positive integer:

performing convolution operation on the specified image based on a preset convolution check to obtain an intermediate image;

Determining the position information of a target pixel point in the intermediate image as the position information of an ith bright spot, wherein the pixel value of the target pixel point is the maximum value of all pixel values in the intermediate image;

wherein, in the case where i is 1, the specified image is the human eye image; in the case where i is greater than 1, the specified image is acquired by: setting a pixel value of a target area in the human eye image as a specified pixel value to obtain the specified image, wherein the target area is determined based on the position information of the first i-1 bright spots.

4. The method of claim 2, wherein the determining shape information of the bright spots based on the bright spot marker image comprises:

and predicting the bright spot marked image through a classification model to obtain the shape information of the bright spot in the bright spot marked image, wherein the classification model is obtained by training an initial model based on at least one training sample, and at least one training sample comprises the bright spot image, and the bright spot image is marked with the real shape information of the bright spot.

5. The method according to any one of claims 1 to 4, wherein determining the area of human eye gaze based on the correspondence between the bright spots and the light sources comprises:

Based on the corresponding relation between the bright spots and the light source, determining the spatial position information of the cornea center, wherein the spatial position information of the cornea center represents the position information of the cornea center in a target space, and the target space is a three-dimensional space in which the head-mounted augmented reality device is positioned;

determining the position information of the pupil center and the contour information of the pupil in the human eye image based on the human eye image;

determining the spatial position information of the pupil center based on the spatial position information of the cornea center, the position information of the pupil center and the contour information of the pupil;

determining a visual axis based on the spatial position information of the cornea center and the spatial position information of the pupil center, the visual axis being used to characterize a gaze direction of a human eye;

based on the visual axis, the human eye gaze area is determined.

6. The method of claim 5, wherein determining the location information of the pupil center and the contour information of the pupil in the human eye image based on the human eye image comprises:

determining an area with a pixel value smaller than or equal to a preset pixel threshold value in the human eye image as a pupil area;

and determining the position information of the pupil center and the contour information of the pupil based on the pupil area.

7. The method of claim 4, wherein the training process of the classification model is as follows:

training an initial model by using a sample data set to obtain the classification model, wherein the sample data set comprises a bright spot image marked with real shape information;

the implementation mode of each model training comprises the following steps:

inputting the bright spot image into the initial model for prediction processing to obtain predicted shape information of the bright spot image;

and adjusting parameters of the initial model based on the relative error and the loss function between the predicted shape information and the marked real shape information, and taking the initial model with the adjusted parameters as an initial model for the next model training.

8. A detection apparatus for an eye gaze area, the apparatus being applied to a head-mounted augmented reality device comprising a plurality of light sources, wherein at least one of the light sources is not identical in shape to the other light sources, the apparatus comprising:

the device comprises an image acquisition module, a display module and a display module, wherein the image acquisition module is used for acquiring a human eye image, the human eye image is acquired from an eye area of a user wearing the head-mounted display device, and the human eye image comprises at least two bright spots;

The first determining module is used for determining the shape information of the bright spots based on the human eye image;

the second determining module is used for determining the corresponding relation between the bright spots and the light source based on the shape information of the bright spots;

and a third determining module, configured to determine a human eye gazing area based on a correspondence between the bright spots and the light source.

9. A head-mounted augmented reality device, comprising:

a plurality of light sources, wherein at least one of the light sources has a shape that is different from the shape of the other light sources;

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method of any of claims 1-7.

10. A computer readable storage medium having stored therein computer program instructions which are callable by a processor to perform the method according to any one of claims 1-7.