KR20010055957A - Image Registration Method Using 3D Tracker And Computer Vision For Augmented Reality - Google Patents

Abstract

The present invention relates to a method of more accurately synthesizing real-life and virtual images, using a three-dimensional tracker and computer vision that can measure position and posture to find the position and posture of the camera more stably than conventional methods In order to provide image matching using augmented reality-based 3D tracker and computer vision, the internal parameters of the camera for capturing real images are determined and a database is constructed for image processing by computer vision. First step; A second step of extracting the position and attitude information of the camera from the tracker while capturing the live image; A third step of correcting the position and attitude of the camera using information extracted from the tracker and feature points obtained through computer vision processing using a database; And a fourth step of synthesizing the virtual image into the real image using the corrected position and posture of the camera, and used in an augmented reality application system.

Description

Image Registration Method Using 3D Tracker And Computer Vision For Augmented Reality

The present invention uses an augmented reality-based three-dimensional tracker and computer vision in augmented reality application system that synthesizes a computer-generated virtual image to a real-life image captured by the camera, more accurately synthesized real-life image and a virtual image An image matching method and a computer-readable recording medium having recorded thereon a program for realizing the method.

In order to provide image expression or information that is impossible in the real world, a lot of researches have recently been conducted on technologies that provide easier and useful information by mixing the real world in which people live and the cyber world generated by computers. Augmented Reality technology.

Augmented reality technology is a field where full-fledged research began in the 1990s, but there are not many commercialized systems, but active research is being conducted recently due to the potential of many applications and technologies.

In order to match the virtual image to the real image in the augmented reality system, the data modeled in three dimensions in advance using the position and attitude values of the camera is used to generate the virtual image using a three-dimensional perspective projection method such as a real camera projecting the real image. After rendering, the real image and the virtual image are synthesized and displayed.

Virtual images projected using the camera's position and posture information change very sensitively to the camera's position and posture error, so it is necessary to know the exact camera position and posture information.

In order to know the position and posture of the camera, a method of using a tracker that can measure a three-dimensional position and posture and a method of obtaining the position and posture of the camera using a feature point known in advance using computer vision have been used. .

However, the method of using a tracker such as a GPS (Global Position System), a magnetic tracker, a gyroscope, and the like does not provide enough precision for image matching due to an error included in the tracker itself. It is necessary to extract markers or feature points necessary for position and posture calculation every frame, and this operation is very sensitive to the surrounding environment, and in some cases, there is a problem in that a feature image cannot be generated because a feature point for image registration cannot be found.

The present invention has been made to solve the above problems, by using a three-dimensional tracker and computer vision that can measure the position and posture, augmented for image matching to find the position and posture of the camera more stably than conventional methods An object of the present invention is to provide an image matching method using a reality-based three-dimensional tracker and a computer vision, and a computer-readable recording medium recording a program for realizing the method.

1 is a configuration diagram of an embodiment of the augmented reality system to which the present invention is applied.

2 is a flowchart illustrating an image matching method using an augmented reality-based 3D tracker and computer vision according to the present invention.

Figure 3 is an exemplary view of a three-dimensional tetrahedron showing a search range for the position and attitude to which the present invention is applied.

Figure 4 is an exemplary view of a calculation method for calculating the accuracy of the camera position and attitude value to which the present invention is applied.

Explanation of symbols on the main parts of the drawings

10: tracker (receiver) 20: tracker (transmitter)

30: tracker processor 40: CCD camera

50: Computer 60: 3D Modeling Data

70: marker or feature point

The present invention for achieving the above object, in the image registration method using a three-dimensional tracker and computer vision applied to the augmented reality system, determining the internal parameters of the camera for capturing the real image, image processing by computer vision A first step of building a database for the purpose; A second step of extracting position and attitude information of the camera from a tracker while capturing a live-action image; A third step of correcting the position and attitude of the camera using the information extracted from the tracker and the feature point obtained through computer vision processing using the database with respect to the live image; And synthesizing a virtual image to the real image using the corrected position and posture values of the camera.

In addition, in order to match images using a 3D tracker and computer vision, an augmented reality system equipped with a large-capacity processor determines internal parameters of a camera for capturing live images, and a database for image processing by computer vision. A first function of building a; A second function of extracting position and attitude information of the camera from a tracker while capturing a live-action image; A third function of correcting the position and attitude of the camera using the information extracted from the tracker and the feature point obtained through computer vision processing using the database with respect to the live image; And a computer readable recording medium having recorded thereon a program for realizing a fourth function of synthesizing a virtual image to the live image using the corrected position and attitude value of the camera.

The present invention uses a combination of the two methods to complement the currently used tracker method and computer vision method in augmented reality system for synthesizing real-life and virtual images, first extract the approximate camera position and pose from the tracker Then, using the 3D feature point database pre-built using computer vision and the feature points obtained through image processing, a detailed search using a genetic algorithm to make detailed corrections so that the difference in the coordinate values displayed on the screen is reduced. Coarse-to-Fine error correction method is used.

That is, in the present invention, by accurately measuring the position and posture of the camera, which is one of the most important issues in implementing the augmented reality system, it is intended to accurately synthesize the actual image captured by the camera and the virtual image generated by the computer.

The operation of the present invention first generates three-dimensional modeling data using computer graphics to generate a virtual image corresponding to the real-world, and attach a marker or feature for easy feature extraction using computer vision in the region where the virtual image overlaps The positions of the markers and the feature positions in the 3D modeling data are stored in separate databases.

In addition, a tracker capable of measuring three-dimensional position and posture is attached to the camera in order to measure the position and posture information of the camera required to generate the virtual information of the actual photo captured by the camera. In addition, the markers are extracted from the live image captured by the camera in order to determine the exact position and posture of the camera based on the measured position and posture data. Fine tune.

This process uses the screen position of the marker captured by the camera and the position and posture value searched based on the position and posture values calculated by the tracker to capture the 3D coordinates of the marker on the screen. A virtual image is generated by perspectively projecting the entire three-dimensional modeling data by selecting the position and attitude of the final camera where the distance difference between the screen coordinate value of the marker and the screen coordinate value of the perspectively projected marker is the smallest. .

Then, it is synthesized with the live image captured by the camera and displayed to the user. Therefore, visually more accurate synthesis can be achieved than the image matching method used in the existing augmented reality system.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to FIGS. 1 to 4.

1 is a configuration diagram of an embodiment of an augmented reality system to which the present invention is applied, and a tracker capable of measuring a three-dimensional position and pose such as a GPS or a magnetic tracker attached to the camera to measure the position and pose of the camera. 10, a CCD camera 40 for capturing live-action, a computer system 50 for processing the acquired information and synthesizing the real-time and virtual information, and displaying the real-world information to generate virtual information superimposed on the captured image. Equally three-dimensionally modeled data 60 and computer vision-based error correction for the three-dimensional data consisting of the markers and feature points 70 previously measured.

2 is a flowchart illustrating an image matching method using an augmented reality-based 3D tracker and a computer vision according to the present invention, which is divided into a calibration step and a system operation step.

In this case, the error (Derr), the maximum allowable value, and the error range used are defined as follows.

The error (Derr) is the distance between the screen coordinate values obtained by perspective projection of the marker's three-dimensional coordinate values using the screen coordinate values of the marker and the position and attitude values searched based on the tracker's position and attitude. Say.

The maximum allowable value refers to a value set by the user in advance to stop searching when the error value (Derr) becomes smaller than the user wants while searching for the position and attitude.

The margin of error is a measure of the tracker's precision from the manufacturer of the tracker. For example, if the positional accuracy of the tracker is less than 1 cm, if the position is measured using the tracker where the actual position is 10 cm away from the reference point of the tracker, the result is within 9 cm to 11 cm. Therefore, the error range referred to here refers to a range of +,-1 cm based on the position calculated by the tracker.

First, in the initial calibration step, the internal parameters of the camera, that is, the focal length and the vertical and horizontal angles of view are measured (201), and a feature point database is constructed by extracting feature points extracted from an image captured by the camera (202). For the parts that are difficult to extract, a process 202 for positioning a marker promised for active feature extraction is performed.

The screen coordinates of the markers captured by the CCD camera and the markers calculated by the tracker to complete the search range, the equal fraction of the step-by-step search points, and the search for detailed position and posture search using the image captured by the CCD camera. The maximum allowable value and the maximum allowable search time for the error consisting of the difference in distance from the screen coordinates of the markers perspectively projected using the position and the attitude value obtained by searching based on the position and attitude of the user It is set in advance (203).

At this time, in order to operate the genetic algorithm, a predetermined number of positions (X, Y, Z) and posture values (Yaw, Roll, Pitch) are randomly set in the search area based on the search range and the equal fraction of the search point in advance. It produces several populations of genes, or populations, of dog chromosomes.

Next, the system operation step extracts the position and posture of the current point of view from the tracker 10 attached to the camera, and captures the live image from the CCD camera 40 (204). That is, the approximate camera position and pose are extracted from the tracker.

Then, the markers that can be seen in the current view are found in the previously constructed marker database using the position and posture information obtained by the tracker 10, and then the dictionary is centered on the screen coordinate points generated by perspectively projecting the corresponding markers onto the screen. A feature point 70 using image processing is extracted with respect to the image having the input range in step 205.

In this case, when the marker is defined in the feature point extraction, the feature point is extracted using a database for the marker.

In the next step, a detailed search is performed for error correction with a feature point within the error range of the tracker's position and attitude based on the position and the attitude information calculated by the tracker (206).

At this time, the detailed search process includes the movement values (X, Y, Z) on the X, Y, and Z axes, which are variables of position and attitude, and the rotation values (yaw, roll, pitch) based on the X, Y, and Z axes. It consists of six variables and the search range for position and posture, as shown in FIG. 3, is based on the respective three-dimensional tetrahedrons.

The process of detailed search for error correction using the feature point and the position and posture of the tracker is a search method using a genetic algorithm for a search range for six variables made around the position and posture of the tracker. Is performed.

That is, the genetic algorithm considers each variable for the variables X, Y, Z, Roll, Pitch, and Yaw to be found as one chromosome, and the position and posture to be searched are referred to as one gene composed of six chromosomes. .

Gene populations generated during the initial calibration stage, ie, populations, are used to crossover each other at a predetermined random variable ratio, and the chromosomes in the gene are pre-designated with random variables. Mutation

Then, the performance of each gene is evaluated using the error (Derr) to select a gene having a good performance, and to select a gene that does not have a good one, and to generate a new gene and add it to the population.

Then, the hybridization and mutation process are performed again. By continuing this process, most genes in the population converge to have the smallest Derr between the feature points.

This process takes some time. Depending on the performance of the system, the actual image is delayed by about 2-3 frames, the gene with the lowest error (Derr) is selected within a given time, and then 3D modeling is performed. The data is synthesized with the images projected by the perspective projection and the images captured by the CCD camera.

In other words, while changing the position and posture within the tracker's error range based on the position and posture calculated by the tracker, the screen coordinate values of the markers projected by the perspective are the same as the screen coordinate values of the marker captured by the CCD camera. After finding a suitable position and posture, a virtual image is generated using the synthesized image (208), and the real image and the virtual image captured by the CCD camera are synthesized (209).

4 is a perspective view of a three-dimensional data using the position and attitude values to which the present invention is applied, and a calculation method for calculating accuracy applied in the present invention. It is a calculation method used in the process (206, 207) for changing the position and attitude by the error (Derr) between the center of the camera.

Perspective projection first converts a point in the world coordinate system (x _w , y _w , z _w ) to a tracker coordinate system based on the tracker, and then plots the diagram with the focal length of the camera and the two-dimensional screen size values. We can get the screen coordinates (x _u , y _u ) by using matrix operation.

The camera coordinate system is a coordinate system composed of the exact position and attitude value for perspective projection to the screen position of the marker obtained from the real computer vision, and the coordinates for the projection of the feature points (x _wn , y _wn , z _wn ) are (x _u ', y _u ').

The tracker coordinate system is a coordinate system for the position and attitude value used to generate the virtual image. The coordinates of the feature point (x, y, z) as a result of perspective projection are referred to as (x _u , y _u ). Define the error (D _err ) for the position and attitude value as the sum of the distances between two corresponding points (x _un ', y _un ') (x _un , y _un ) for each feature point.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the spirit of the present invention for those skilled in the art to which the present invention pertains, and the above-described embodiments and accompanying It is not limited to the drawing.

As described above, the present invention measures the position and posture of the camera essential for implementing the augmented reality system, using the 3D tracker and the computer vision-based pros and cons, and calculates the approximate position of the camera with the position and posture calculated by the tracker. Using Coarse-to-fine search method that calculates position and posture, extracts feature points for local computer vision by using tracker error range, and finely corrects camera position and posture using it Therefore, it is possible to extract the position and posture more accurate than the existing method, and also has the excellent effect that the implementation of the augmented reality system that can provide the user with more realistic synthesized photorealistic and virtual information.

In addition, since the two tracking systems are mixed to use the tracker and the computer vision independently, the advantages and disadvantages of each tracking system are compensated for, and thus the stability of the augmented reality system is increased.

Claims (5)

In the image registration method using 3D tracker and computer vision applied to augmented reality system, A first step of determining an internal parameter of a camera for capturing a live image, and constructing a database for image processing by computer vision; A second step of extracting position and attitude information of the camera from a tracker while capturing a live-action image; A third step of correcting the position and attitude of the camera using the information extracted from the tracker and the feature point obtained through computer vision processing using the database with respect to the live image; And A fourth step of synthesizing the virtual image with the live image using the corrected position and posture of the camera Image matching method comprising a. The method of claim 1, A fifth step of defining a marker and constructing a database for the marker when it is difficult to construct a database for the feature point Image matching method further comprising. The method of claim 2, And the feature point of the third step is extracted using a database for the marker. The method according to any one of claims 1 to 3, The image registration method, characterized in that for correcting the position and posture of the camera using a genetic algorithm. In the augmented reality system equipped with a high-capacity processor, to match the image using a three-dimensional tracker and computer vision, A first function of determining internal parameters of a camera for capturing live-action images, and constructing a database for image processing by computer vision; A second function of extracting position and attitude information of the camera from a tracker while capturing a live-action image; A third function of correcting the position and attitude of the camera using the information extracted from the tracker and the feature point obtained through computer vision processing using the database with respect to the live image; And A fourth function of synthesizing a virtual image to the real image using the corrected position and posture of the camera A computer-readable recording medium having recorded thereon a program for realizing this.