JP5127165B2 - Information processing method and apparatus - Google Patents

Description

  The present invention relates to a technique for identifying an index in an image.

[Prior art 1]
Position / orientation measurement of an imaging unit such as a camera that captures a real space (hereinafter referred to as a camera as appropriate) is required in, for example, a mixed reality system that displays a fusion of real space and virtual space. As such a conventional technique, the position and orientation of the camera is determined by using a marker whose position in the real space is known or a feature point whose position in the real space is known (hereinafter referred to as an index by combining the marker and the feature point). There is a method of correcting a measurement error of a position / orientation sensor that measures the angle. These methods are disclosed in Patent Documents 1 to 2 and Non-Patent Document 1.

[Prior Art 2]
On the other hand, as disclosed in Non-Patent Documents 2 to 3, there is also known a method for estimating the position and orientation of a camera using only an index captured by the camera without using a position and orientation sensor. In these non-patent documents, the position and orientation of the camera are estimated based on the coordinates of the four vertices of the square using a square index. However, since a square is a rotationally symmetric shape every 90 ° with the axis perpendicular to the plane passing through its center point (intersection of diagonal lines), the top / bottom / left / right determination can be made only from the coordinates of the vertex. Can not. For this reason, a further image feature for performing up / down / left / right determination is provided inside the square index. In addition, when using multiple indicators, it is necessary to identify which of the multiple indicators is captured based only on the image captured by the camera. And graphic information such as codes are embedded.

[Prior Art 3]
In addition, a method is known in which a plurality of imaging units with known positions and orientations arranged in the real space are used to image a plurality of indices whose positions on the target are known to estimate the position and orientation of the target. In the prior art 3, as a means for determining which of a plurality of indices the detected index corresponds to, the light emission timing is controlled for each LED serving as the index.
Japanese Patent Laid-Open No. 11-084307 JP 2000-041173 A JP 2004-233334 A A. State, G. Hirota, D.H. T.A. Chen, B.M. Garrett, and M.M. Livingston: Superior augmented real estate registration by integrating landmark tracking and magnetic tracking, Proc. SIGGRAPH '96, pp. 429-438, July 1996. Kato, Billinghurst, Asano, Tachibana: Augmented reality system based on marker tracking and its calibration, Transactions of the Virtual Reality Society of Japan, vol. 4, no. 4, pp. 607-616, Dec. 1999. X. Zhang, S.M. Fronz, N.A. Navab: Visualmarker detection and decoding in AR systems: A comparative study, Proc. of International Symposium on Mixed and Augmented Reality (ISMAL'02), 2002. R. M.M. Haralick, C.I. Lee, K.M. Ottenberg, and M.M. Nole: Review and analysis of solutions of the three point perspective positive estimation lab, International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994. D. G. Low: Fitting parametrized three-dimensional models to images, IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-450, 1991. Hirofumi Fujii, Masayuki Kanbara, Hidehiko Iwasa, Haruo Takemura, Naokazu Yokoya, alignment with stereo camera combined with gyro sensor for augmented reality, IEICE Technical Report PRMU99-192 , No. 574, pp. 1-8) H. Najafi, N .; Navab, G .; Klinker: Automated Initialization For Marker-less Tracking: A Sensor Fusion Approach, Proc. of International Symposium on Mixed and Augmented Reality, pp. 79-88, 2004.

  In the method of estimating the position and orientation of the camera according to the related art 1, a small circular sheet-like object having a specific color can be used as an index. In this case, information of the index is a three-dimensional position (coordinates) and a color. Using the measured values of the position and orientation sensor, the three-dimensional position of the index is projected onto the image plane of the camera, while color area detection processing is performed to detect the color of the index from the image, and the center of gravity in the image Calculate the position. Then, the index in the image can be identified by comparing the three-dimensional position projected onto the image plane with the barycentric position calculated from the image and determining that the closest one is the same index, for example.

  In this way, when detecting an index from an image by color region detection, if the same color as the index exists in the real space that the camera will capture, other than the index, they are erroneously identified. There's a problem.

  In order to prevent such a problem, there is a method of detecting only a correct combination area as an index by using an index composed of a combination of different colors arranged concentrically and examining a color combination after performing color area detection. . In this case, it is less likely that a part of the background is erroneously detected as an index, compared to the case where a single color index is used.

  However, since stable index detection is performed using color area detection, the index color is often set to a conspicuous color. Furthermore, when combining different colors on the concentric circles, the image must be captured sufficiently large in the image in order to stably detect the concentric circles. In other words, it is necessary to arrange an indicator that is large and impairs the appearance in the real space. However, there are cases where it is not allowed to place such an indicator in the real space, and there is room for improvement in terms of deteriorating the appearance of the real space.

  On the other hand, there is a method of using an index having a graphic expansion such as a square marker used in the prior art 2. However, in the prior art 2, since it is necessary to identify each marker completely only from an image, it is necessary to embed code information, symbol information that can be a template, or the like in an index. FIG. 9 is an example of a specific square marker used in the related art 2 disclosed in Non-Patent Document 2 and Non-Patent Document 3.

  Since an index having such a complicated structure must be detected from the captured image, there is a problem that the index cannot be recognized unless the index is captured so as to occupy a sufficiently large area in the captured image plane. . In other words, this means that a large area of the real space has to be reserved for indicator placement, or that the camera has to be close enough to the indicator. In other words, it can be paraphrased as a problem that the arrangement condition of the index is severe.

  In order to identify individual indices while reducing the indices so as to be inconspicuous, there is a method of using a light emitter such as an LED or a retroreflecting material as an index. However, such a method has a problem that when a light emitter or a reflector is present in addition to the index, they are erroneously determined as the index.

  For example, as shown in FIG. 10, a case where a light emitter is present in addition to an index in a background space will be described as an example. In FIG. 10, it is assumed that an index 205 to be used is arranged on a real object 204, and is, for example, a sphere that emits infrared light. Further, it is assumed that infrared light is emitted from a light emitter 203 such as a light bulb. Assume that the image captured by the infrared camera 101 through the visible light cut (infrared light transmission) filter 202 in this state is an image as shown in FIG.

  At this time, if a brightness area detection process for detecting an area corresponding to the brightness of the index 205 from the image is performed, not only the area 301 corresponding to the index 205 but also the area 302 corresponding to the light emitter 203 has the brightness of the index 205. There is a possibility of being detected as a corresponding region. In such a case, there is a problem that it cannot be determined whether the index 205 corresponds to the region 301 or the region 302. As described above, when an object having the same or similar brightness as the index exists in the imaged space, there is a problem that it may be erroneously recognized as the index.

  In order to identify an index using a light emitter or reflector, there is a method in which several light emitters or reflectors are arranged with their relative positions fixed, and these are used together as an index. In this method, individual indices are identified using the positional relationship between the light emitters or reflectors. However, there is a problem that if some of the light emitters or reflectors are hidden, they cannot be identified as an index. Moreover, since the index is large, there is a problem that the appearance of the real space is deteriorated.

  On the other hand, in the prior art 3, in order to identify a plurality of indices, an index capable of controlling the light emission timing in a time division manner is used. There was a problem of cost.

  The present invention has been made to solve such problems of the prior art, and an object of the present invention is to provide an information processing method and apparatus capable of correctly identifying an index.

  In order to achieve the object of the present invention, for example, an information processing method and apparatus of the present invention comprises the following arrangement.

The information processing method according to the present invention includes a first objective image input step of inputting a first image taken by an imaging device for imaging a scene, and stereo objective viewpoint imaging from a plurality of objective viewpoint positions to which the imaging device is fixed. A second image input step of inputting a plurality of second images imaged by the means, and an inclination angle input step of inputting an inclination angle measurement value from an inclination angle measurement device for measuring information relating to the inclination angle of the imaging device A feature value of the first index in the scene is detected from a provisional setting step of provisionally setting a plurality of azimuth angles of the imaging device and a first image input in the first image input step. A first detection step and a second detection step of detecting a feature quantity of a candidate for a second index included in the imaging device from each of a plurality of second images input in the second image input step. If, for each each of the plurality of azimuth it said has been tentatively set Based on the first index position calculation step of calculating a plurality of projection positions of the first index and the feature amount of the first index detected in the first detection step, the plurality of first indices By selecting one of the projection positions, the first index of the detected feature amount is identified as indicating the first index of the selected projection position, and the selected projection position Based on the first identification step in which the tentatively set azimuth corresponding to the calculation of azimuth is the azimuth of the imaging device and the feature quantity detected in the first detection step, the second index Based on the second index position calculating step for calculating the three-dimensional position and the feature amount detected from each of the plurality of second images in the second detecting step, 3 of the second index candidates a second candidate position calculating step of calculating a dimension position, the second Based on three-dimensional distance between the three-dimensional position of the three-dimensional position of the index candidate, the candidate is characterized by having a second identification step of identifying whether indicating the second index .

The information processing apparatus according to the present invention includes a first objective image input unit that inputs a first image captured by an imaging apparatus that captures a scene, and stereo objective viewpoint imaging from a plurality of objective viewpoint positions to which the imaging apparatus is fixed. A second image input means for inputting a plurality of second images picked up by the means, and an inclination angle input means for inputting an inclination angle measurement value from an inclination angle measurement apparatus for measuring information relating to the inclination angle of the image pickup apparatus. And a temporary setting means for temporarily setting a plurality of azimuth angles of the imaging device, and a feature value of the first index in the scene is detected from the first image input by the first image input means. Second detection means for detecting a feature quantity of a candidate for a second index included in the imaging device from each of a plurality of second images input by the first detection means and the second image input means. If, for each each of the plurality of azimuth it said has been tentatively set Based on the first index position calculation means for calculating a plurality of projection positions of the first index and the feature quantity of the first index detected by the first detection means, the plurality of first index By selecting one of the projection positions, the first index of the detected feature amount is identified as indicating the first index of the selected projection position, and the selected projection position Based on the first identification means that uses the temporarily set azimuth corresponding to the calculation of azimuth as the azimuth angle of the imaging device, and the feature quantity detected by the first detection means, the second index Based on the second index position calculating means for calculating the three-dimensional position and the feature quantity detected from each of the plurality of second images by the second detecting means, 3 of the second index candidates a second candidate position calculating means for calculating a dimension position, before Based on three-dimensional distance between the three-dimensional position of the candidate and the three-dimensional position of the second indicator, that the candidate and a second identification means for identifying whether indicating the second index A characteristic information processing apparatus.

  According to the present invention, when associating an index with an index candidate area detected from an image, an area similar to the index exists in the real space, and the area is detected as an index candidate area. Even in such a case, the index can be correctly identified. Further, by using the identified index, the position and orientation of the imaging device or the target object can be calculated with higher accuracy.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
The information processing apparatus according to the present embodiment identifies an index based on the estimated position of the imaging apparatus.

  FIG. 1 shows the configuration of the information processing apparatus in the present embodiment. The information processing apparatus 100 according to the present embodiment includes an image input unit 160, a data storage unit 170, a subjective viewpoint index detection unit 110, a position and orientation estimation unit 120, an objective viewpoint index candidate detection unit 140, a position constraint condition calculation unit 150, and an objective. The viewpoint index identification unit 190 is configured. The information processing apparatus 100 is connected to the objective viewpoint camera 180 and the imaging apparatus 130. The position of the objective viewpoint camera 180 is fixed, and the imaging device 130 is, for example, a head-mounted device that can be mounted on the operator's head and moved.

At a plurality of positions in the real space, the world coordinate system (one point in the real space is defined as the origin, and three axes that are orthogonal to each other are used as indices for photographing with the imaging device 130 (hereinafter referred to as subjective viewpoint indices). A plurality of subjective viewpoint indices Q _k (k = 1,..., K _Q ) having known positions x _W ^Qk in the coordinate system defined as the X axis, Y axis, and Z axis, respectively, are arranged. Further, on the imaging device 130, an index for imaging with the objective viewpoint camera 180 (hereinafter referred to as objective viewpoint index) is defined as an imaging device coordinate system (one point on the imaging device 130 is defined as an origin, and further orthogonal to each other). the three axes respectively X-axis, Y-axis, an objective-view-index P position x _C ^P in the defined coordinate system) on the Z axis is known has been set.

When the imaging device 130 is positioned at each of a plurality of points in the field of view of the objective viewpoint camera 180, at least three subjective viewpoint indices _Qk are always observed on the subjective viewpoint image acquired by the imaging device 130. Thus, it is desirable that the imaging device 130 be positioned. In addition, it is desirable that the imaging device 130 be positioned so that the objective viewpoint index P is always observed on the objective viewpoint image acquired by the objective viewpoint camera 180. In the example of FIG. 1, four subjective viewpoint indices Q ₁ , Q ₂ , Q ₃ , Q ₄ and one objective viewpoint index P are set, and a false index P ′ similar to the objective viewpoint index P is set. Exist in the real space, and three subjective viewpoint indices Q ₁ , Q ₃ , and Q ₄ are included in the field of view of the imaging device 130, and one objective viewpoint index P and one objective viewpoint index P This shows a situation where the false index P ′ is included in the field of view of the objective viewpoint camera 180.

Note that the subjective viewpoint index Q ^k may be constituted by, for example, circular markers each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a rectangular single color area having a certain area. Any form may be used as long as the image coordinates of the projected image on the photographed image can be detected and the index can be identified by any method. The subjective viewpoint index may be set by the operator, or may be a natural shape that is not set by the operator.

  On the other hand, since the objective viewpoint index P needs to be stably detected on an image photographed from a distance with a wide-angle objective viewpoint camera, it is composed of, for example, an LED or a retroreflecting material.

  An image output from the imaging device 130 (hereinafter referred to as a subjective viewpoint image) is input to the image input unit 160. An image output from the objective viewpoint camera 180 (hereinafter referred to as an objective viewpoint image) is also input to the image input unit 160.

  The objective viewpoint camera 180 is arranged with the objective viewpoint index P on the imaging device 130 fixed at a position where it can always be imaged. Here, it is assumed that the position and orientation of the objective viewpoint camera 180 in the world coordinate system are held in the data storage unit 170 in advance as known values.

  The image input unit 160 converts the subjective viewpoint image and the objective viewpoint image input to the information processing apparatus 100 into digital data and stores the digital data in the data storage unit 170.

  The subjective viewpoint index detection unit 110 inputs a subjective viewpoint image from the data storage unit 170 and detects image coordinates of the subjective viewpoint index captured in the input image. For example, when each subjective viewpoint index is composed of markers having different colors, an area corresponding to each marker color is detected from the image, and the position of the center of gravity is set as the detection coordinate of the subjective viewpoint index. When each subjective viewpoint index is composed of feature points having different texture features, template matching is performed on the image using the template image of each subjective viewpoint index previously stored as known information. Thus, the position of the subjective viewpoint index is detected. In addition, when a quadratic index is used, labeling is performed after binarizing the image, and an area formed by four straight lines is detected as an index candidate. Further, by detecting whether or not there is a specific pattern in the candidate area, false detection is eliminated, and the identifier of the index is acquired. In this embodiment, the quadrangular indices detected in this way are considered to be four indices specified by four vertices. Note that an estimated value of the position of the imaging device 130 (an output of a position / orientation estimation unit 120 described later) is further input from the data storage unit 170, and the position of the subjective viewpoint index on the image is predicted based on this estimated value to search the range. By limiting this, it is possible to reduce the calculation load of the subjective viewpoint index detection process, and to reduce the erroneous detection and erroneous identification of the subjective viewpoint index.

The subjective viewpoint index detection unit 110 further outputs the image coordinates of the detected index and the identifier of the index to the data storage unit 170. In the following, an index detected on the subjective viewpoint image is denoted as Q _kn using identifiers n (n = 1,..., N) attached to the detected indices. Here, N represents the number of indices detected on the subjective viewpoint image. Further, the image coordinate of the detected index Q _kn is expressed as u ^Qkn . For example, in the case of FIG. 1, N = 3, index identifiers k ₁ = 1, k ₂ = 3, k ₃ = 4, and image coordinates u ^{Qk 1} , u ^{Qk 2} , u ^Qk ₃ corresponding to these are output. The

The position / orientation estimation unit 120 uses a data storage unit 170 to set a set of image coordinates u ^Qkn of each subjective viewpoint index detected by the subjective viewpoint index detection unit 110 and world coordinates x _W ^Qkn held in advance as known information. And the position and orientation of the imaging device 130 are calculated (estimated) based on the information. Estimated position and orientation, for example, is output to the data storage unit 170 by a set of forms of the 3 × 3 matrix R _WC representing a three-dimensional vector x _W ^C and attitude representing the position. A method for calculating the position and orientation of the imaging device 130 from a set of world coordinates and image coordinates of the subjective viewpoint index is known in the field of photogrammetry and the like (for example, see Non-Patent Document 4 and Non-Patent Document 5). Therefore, the details are omitted.

  The objective viewpoint index candidate detection unit 140 inputs an objective viewpoint image from the data storage unit 170 and detects image coordinates of an objective viewpoint index (and a false index) photographed in the input objective viewpoint image. For example, an area corresponding to brightness, such as an LED or a retroreflecting material, is detected on the image, and the position of the center of gravity is used as the detection coordinate of the objective viewpoint index.

The objective viewpoint index candidate detection unit 140 further outputs the image coordinates of the detected objective viewpoint index candidate to the data storage unit 170. In the following description, the objective viewpoint index candidates detected on the objective viewpoint image are represented as P _m by using identifiers m (m = 1,, M) attached to the detected objective viewpoint index candidates. write. Here, M represents the number of objective viewpoint index candidates detected on the objective viewpoint image. Also, the image coordinates of the detected objective index candidate P _m is expressed as u ^Pm. For example, in the case of FIG. 1, M = 2, and image coordinates u ^P1 and u ^P2 are output.

The position constraint condition calculation unit 150 inputs the image coordinates u ^{Pm of} each objective viewpoint index candidate detected by the objective viewpoint index candidate detection unit 140 and the position and orientation of the objective viewpoint camera in the world coordinates from the data storage unit 170. Based on these pieces of information, the position constraint conditions of each objective viewpoint index candidate (in this embodiment, a parameter describing a straight line that restrains the position) are calculated.

  The objective viewpoint index identification unit 190 is based on the position and orientation of the imaging apparatus estimated by the position / orientation estimation unit 120 and the position constraint condition of each objective viewpoint index candidate calculated by the position constraint condition calculation unit 150. Identify objective viewpoint indicators.

  The data storage unit 170 includes an image input from the image input unit 160, an estimated value of a position input from the position and orientation estimation unit 120, an image coordinate and an identifier of each index input from the subjective viewpoint index detection unit 110, an objective The image coordinates of each index candidate input from the viewpoint index candidate detection unit 140, the world coordinates of the subjective viewpoint index, which are known values, the imaging device coordinates of the objective viewpoint index (coordinate values in the imaging device coordinate system), the objective Data such as camera parameters of the viewpoint camera 180 is held, and these are input / output as necessary.

  In addition, the image input unit 160, the data storage unit 170, the subjective viewpoint index detection unit 110, the position and orientation estimation unit 120, the objective viewpoint index candidate detection unit 140, the position constraint condition calculation unit 150, and the objective viewpoint index identification illustrated in FIG. Each of the units 190 may be handled as an independent device, or may be installed in one or a plurality of computers as software and executed by a CPU of each computer to realize the function. . In this embodiment, each unit (image input unit 160, data storage unit 170, subjective viewpoint index detection unit 110, position and orientation estimation unit 120, objective viewpoint index candidate detection unit 140, position constraint condition calculation unit 150, and objective viewpoint index identification) The unit 190) is treated as a function realized by executing software to be executed in one computer.

  FIG. 2 illustrates an image input unit 160, a data storage unit 170, a subjective viewpoint index detection unit 110, a position / posture estimation unit 120, an objective viewpoint index candidate detection unit 140, a position constraint condition calculation unit 150, and an objective viewpoint index identification unit 190. FIG.

  A CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003, and also includes an image input unit 160, a data storage unit 170, a subjective viewpoint index detection unit 110, a position and orientation estimation unit 120, an objective. The execution of each software of the viewpoint index candidate detection unit 140, the position constraint condition calculation unit 150, and the objective viewpoint index identification unit 190 is controlled to realize the function of each unit.

  A RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and also includes a work area necessary for the CPU 1001 to perform various processes. The function of the data storage unit 170 is realized by the RAM 1002.

  A ROM 1003 generally stores a computer storage program, setting data, and the like. Reference numerals 1004 and 1005 denote a keyboard and a mouse, respectively. An operator can input various instructions to the CPU 1001 using the keyboard and the mouse, respectively.

  A display unit 1006 includes a CRT, a liquid crystal screen, and the like, and can display a message to be displayed for measuring the position and orientation of the imaging device 130, for example.

  Reference numeral 1007 denotes an external storage device that functions as a large-capacity information storage device such as a hard disk, and stores an OS (Operating System), software programs, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  Reference numeral 1008 denotes a storage medium drive, which reads programs and data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with instructions from the CPU 1001 and outputs them to the RAM 1002 or the external storage device 1007.

  Reference numeral 1009 denotes an I / F, an analog video port for connecting the objective viewpoint camera 180 and the imaging device 130 or a digital input / output port such as IEEE 1394, and an Ethernet (registered trademark) for outputting information related to the identified index to the outside. ) Consists of ports and the like. The data input by each is taken into the RAM 1002 via the I / F 1009. A part of the function of the image input unit 160 is realized by the I / F 1009.

  A bus 1010 connects the above-described units.

Figure 3 is a flowchart of a process for calculating a straight line constraining the position of the objective viewpoint index candidate P _m, CPU 1001 functions as a position constraint condition calculating section 150 by executing software programs. It is assumed that the program code according to the flowchart is already loaded in the RAM 1002 in the previous stage of performing the following processing.

In step S4000, the position constraint condition calculating section 150 inputs the image coordinates ^{u Pm} of the detected respective objective viewpoint index candidate _{P m} in an objective-view-index candidate detecting section 140 from the data storage unit 170.

In step S4010, the position constraint condition calculating section 150, based on the image coordinates u ^Pm, calculates a parameter representing a straight line constraining the position of the objective viewpoint index candidate P _m each in the world coordinate system. That is, the slopes h _xm , h _ym , h _zm of the straight line in the world coordinate system are

_Is calculated based on the image coordinates u ^Pm, and h _xm , h _ym , and h _zm are used as linear parameters. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 180 in the x-axis direction and the y-axis direction, respectively, and are held in the data storage unit 170 in advance as known values. _RWB is a rotation matrix that converts the posture in the objective viewpoint camera coordinate system into world coordinates, and is based on the posture of the objective viewpoint camera 180 in the world coordinate system that is stored in advance in the data storage unit 170 as a known value. Suppose that it is calculated in advance. At this time, the point on the straight line in the world coordinate system is a function of the parameter τ.

It can be expressed as Here, x _WB , y _WB , and z _WB are positions of the objective viewpoint camera 180 in the world coordinate system, and are held in the data storage unit 170 in advance as known values.

In step S4020, the position constraint condition calculation unit 150 outputs the slopes h _xm , h _ym , and h _zm of the straight line in the world coordinate system of each objective viewpoint index candidate P _m to the data storage unit 170.

Thus a straight line represented by the formula (2) is a straight line passing through the position of the objective viewpoint index candidate P _m in position and the world coordinate system of the objective viewpoint camera 180 in the world coordinate system, the objective viewpoint indices in the world coordinate system position of the candidate P _m is for the parametric τ is obtained by taking an appropriate value.

  FIG. 4 is a flowchart of processing for identifying an objective viewpoint index, and the CPU 1001 functions as the objective viewpoint index identification unit 190 by executing a software program. The process according to the flowchart is performed by performing the process of step S5010 to step S5030 on each objective viewpoint index candidate after performing the process of step S5000. It is assumed that the program code according to the flowchart is already loaded in the RAM 1002 in the previous stage of performing the following processing.

In step S5000, the objective viewpoint index identifying section 190 inputs the 3 × 3 matrix _{R WC} representing a three-dimensional vector _x ^{W C} and attitude representing the position of the imaging device estimated by the position and orientation estimation unit 120 from the data storage unit 170 Based on this, the estimated position of the objective viewpoint index P is calculated. In other words, the estimated position x _W ^P of the objective viewpoint index P in the world coordinate system,

Calculated by Here, x _C ^P is the coordinate value of the objective-view-index P in the imaging device coordinate system, are held in advance as known information in the data storage unit 170.

When the position of the objective viewpoint index P is sufficiently close to the origin of the imaging apparatus coordinate system, the estimated position of the imaging apparatus may be used as the estimated position of the objective viewpoint index P without performing this step. Good. In that case, the position / orientation estimation unit 120 may not hold the 3 × 3 matrix R _WC representing the orientation of the imaging apparatus in the data storage unit 170.

In step S5010, the objective viewpoint index identifying section 190 calculates the distance to the straight line _{l Wm} (tau) for restraining the position of the objective viewpoint index candidate _{P m} of interest from the estimated position _x ^{W P} objective-view-index P. That is, the following expression for the distance _{d m,}

Is calculated by obtaining the minimum value of the right-hand side (a quadratic function of τ) and taking the positive square root of the value.

In step S5020, the objective viewpoint index identifying section 190, the distance _{d m} obtained in step S5010 it is determined whether within a predetermined range.

  If it is within the predetermined range, the process proceeds to step S5030. In step S5030, the objective viewpoint index identification unit 190 determines that the objective viewpoint index candidate is an objective viewpoint index. For example, the image coordinates of the objective viewpoint index are set to I. Output to the outside via / F1009, and this processing is terminated.

On the other hand, the process advances to step S5040 to the case outside the predetermined range, in step S5040, the objective viewpoint index identifying section 190, for all the objective viewpoint index candidate _{P m} whether or not subjected to the process of step S5010~S5030 to decide. Than is the process ends if you are done, but performed the process returns if not not to a be a step S5010, the objective viewpoint index candidate P _m that has not yet been a processing target step S5010 and subsequent steps S5010 and later Process.

  The objective viewpoint index can be identified by the above processing.

  In the present embodiment, it is assumed that there is one imaging device (one objective viewpoint index), but it goes without saying that there may be a plurality of imaging devices (multiple objective viewpoint indices). Yes. That is, by performing the processing of steps S5000 to S5040 on each objective viewpoint index P, it is only necessary to identify which objective viewpoint index the objective viewpoint index candidate detected from the objective viewpoint image corresponds to.

  As described above, according to the information processing apparatus and the information processing method according to the present embodiment, the light emission timing control mechanism and the high-speed camera that are used in [Prior Art 3] are not required, and as a result, the cost is reduced. Can be kept low.

[Second Embodiment]
The information processing apparatus according to the present embodiment is configured by adding another objective viewpoint camera to the information processing apparatus of the first embodiment.

  FIG. 5 is a diagram illustrating a configuration of the information processing apparatus according to the present embodiment. The same parts as those in FIG. 1 are given the same numbers and symbols, and the description thereof is omitted. As illustrated in FIG. 5, the information processing apparatus 600 according to the present embodiment includes an image input unit 660, a data storage unit 170, a subjective viewpoint index detection unit 110, a position / posture estimation unit 120, an objective viewpoint index candidate detection unit 640, a position The constraint condition calculation unit 650 and the objective viewpoint index identification unit 690 are configured. The information processing apparatus 600 is connected to the objective viewpoint cameras 180a and 180b and the imaging apparatus 130 that is a measurement target. Note that the positions of the objective viewpoint cameras 180a and 180b are fixed, and the imaging device 130 is, for example, a head-mounted device and can be mounted on the operator's head and moved.

A subjective viewpoint index _Qk is arranged at a plurality of positions in the real space, as in the first embodiment. An objective viewpoint index P whose position on the imaging device coordinate system is known is arranged on the imaging device 130.

When the imaging device 130 is positioned at each of a plurality of points in the field of view of the objective viewpoint cameras 180a and 180b, at least three or more subjective viewpoint indices _Qk are always observed on the subjective viewpoint image acquired by the imaging device 130. It is desirable to position the imaging device 130 as described above. In addition, it is desirable that the imaging device 130 is positioned so that the objective viewpoint index P is always observed on the objective viewpoint images acquired by each of the objective viewpoint cameras 180a and 180b. In the example of FIG. 5, four subjective viewpoint indices Q ₁ , Q ₂ , Q ₃ , Q ₄ and one objective viewpoint index P are set, and a false index P ′ similar to the objective viewpoint index P is set. Exist in the real space, and three subjective viewpoint indices Q ₁ , Q ₃ , and Q ₄ are included in the field of view of the imaging device 130, and one objective viewpoint index P and one objective viewpoint index P This shows a situation where the false index P ′ is included in the field of view of the objective viewpoint cameras 180a and 180b.

  The objective viewpoint cameras 180a and 180b are arranged in such a manner that the objective viewpoint index P on the imaging device 130 is always fixed at a position where it can be imaged. Here, it is assumed that the positions and orientations of the objective viewpoint cameras 180a and 180b in the world coordinate system are held in the data storage unit 170 in advance as known values.

  The image input unit 660 converts the subjective viewpoint image and two objective viewpoint images (taken by the objective viewpoint cameras 180 a and 180 b) input to the information processing apparatus 600 into digital data and stores them in the data storage unit 170. .

The objective viewpoint index candidate detection unit 640 receives two objective viewpoint images from the data storage unit 170, detects the image coordinates of the objective viewpoint index candidate, and outputs each image coordinate to the data storage unit 170. . In the following, the objective viewpoint index candidates detected on the objective viewpoint image captured by the objective viewpoint camera 180a are identified by identifiers m _a (m _a = 1,...) Assigned to the detected objective viewpoint index candidates. , M _a ) and expressed as P ^a _ma . Here, M _a represents a number of objective viewpoint index candidate detected on the objective viewpoint image captured by the objective viewpoint camera 180a. Also, the image coordinates of the detected objective index candidate ^P _{a ma} denoted as ^{u Pama.} On the other hand, the objective viewpoint index candidates detected on the objective viewpoint image photographed by the objective viewpoint camera 180b are assigned identifiers m _b (m _b = 1,, M _b) attached to the detected objective viewpoint index candidates. ) And expressed as P ^b _mb . Here, M _b represents the number of objective viewpoint index candidates detected on the objective viewpoint image taken by the objective viewpoint camera 180b. Then, the detected image coordinates of the objective index candidate P ^b _mb are denoted as u ^Pbmb . For example, in the case of FIG. 5, M _a = M _b = 2 and image coordinates u ^Pa1a , u ^Pa2a , u ^Pb1b , u ^Pb2b are output.

The position constraint condition calculation unit 650 includes image coordinates (u ^Pa1a , u ^Pa2a , u ^Pb1b , u ^{Pb2b in} the example of FIG. 5) of each objective viewpoint index candidate detected by the objective viewpoint index candidate detection unit 640, and each of them. The position and orientation of the objective viewpoint camera in the world coordinates are input from the data storage unit 170, and the position constraint condition (three-dimensional position in the present embodiment) of each objective viewpoint index candidate is calculated based on such information.

  The objective viewpoint index identification unit 690 is based on the position of the imaging device 130 estimated by the position and orientation estimation unit 120 and the position constraint condition of each objective viewpoint index candidate calculated by the position constraint condition calculation unit 650. Identify viewpoint indicators.

  Note that the image input unit 660, the data storage unit 170, the subjective viewpoint index detection unit 110, the position and orientation estimation unit 120, the objective viewpoint index candidate detection unit 640, the position constraint condition calculation unit 650, and the objective viewpoint index identification illustrated in FIG. Each of the units 690 may be handled as an independent device, or may be installed in one or a plurality of computers as software and executed by a CPU of each computer to realize the function. In this embodiment, each unit (image input unit 660, data storage unit 170, subjective viewpoint index detection unit 110, position and orientation estimation unit 120, objective viewpoint index candidate detection unit 640, position constraint condition calculation unit 650, and objective viewpoint index identification) Section 690) is treated as a function realized by executing software to be executed in one computer. The basic configuration of this computer is the configuration shown in FIG.

Figure 6 is a flowchart of a process for calculating the three-dimensional position of the objective viewpoint index candidate P _m, CPU 1001 functions as a position constraint condition calculating section 650 by executing software programs. It is assumed that the program code according to the flowchart is already loaded in the RAM 1002 in the previous stage of performing the following processing.

In step S7000, the position constraint condition calculation unit 650 determines the image coordinates (P ^a _ma and P ^b _{mb in} the example of FIG. 5) of the objective viewpoint index candidate detection unit 640 (P ^a _ma and P ^b _{mb in} FIG. 5). In the example, u ^Pa1a , u ^Pa2a , u ^Pb1b , u ^Pb2b ) are input from the data storage unit 170.

In step S <b> 7010, the position constraint condition calculation unit 650 associates objective viewpoint index candidates between images using epipolar constraints, and associates objective viewpoint index candidates P _m (based on the principle of triangulation. The three-dimensional position x _W ^Pm of the objective viewpoint index candidate P _m in the world coordinate system is calculated using the image coordinates of m = 1,. Here, M represents the number of objective viewpoint index candidates associated with each other.

In step S7020, the position constraint condition calculating section 650 outputs the three-dimensional position _x ^{W Pm} of objective viewpoint index candidate _{P m} in the world coordinate system to the data storage unit 170.

  FIG. 7 is a flowchart of processing for identifying an objective viewpoint index, and the CPU 1001 functions as the objective viewpoint index identification unit 690 by executing a software program. The processing according to the flowchart is performed by performing the processing of step S8010 to step S8030 for each objective viewpoint index candidate after performing the processing of step S8000. It is assumed that the program code according to the flowchart is already loaded in the RAM 1002 in the previous stage of performing the following processing.

In step S8000, the objective viewpoint index identifying section 690 inputs the 3 × 3 matrix _{R WC} representing a three-dimensional vector _x ^{W C} and attitude representing the position of the imaging device estimated by the position and orientation estimation unit 120 from the data storage unit 170 Based on Expression 3, the estimated position x _WP of the objective viewpoint index ^P is calculated.

In step S8010, the objective viewpoint index identifying section 690 calculates the distance between the three-dimensional position _x ^{W Pm} of objective viewpoint index candidate _{P m} of interest and the estimated position _x ^{W P} objective-view-index P. That is, the distance _{d m,}

Calculated by

In step S8020, the objective viewpoint index identifying section 690, the distance _{d m} obtained in step S8010 it is determined whether within a predetermined range.

  If it is within the predetermined range, the process proceeds to step S8030. In step S8030, the objective viewpoint index identification unit 690 determines that the objective viewpoint index candidate is an objective viewpoint index. For example, the image coordinates of the objective viewpoint index are set to I. Output to the outside via / F1009, and this processing is terminated.

On the other hand, the process advances to step S8040 to the case outside the predetermined range, in step S8040, the objective viewpoint index identifying section 690, for all the objective viewpoint index candidate _{P m} whether or not subjected to the process of step S8010~S8030 determination and, although of terminating the of value, if the process is being performed, performing the process returns if not not to a be a step S8010, steps for objective viewpoint index candidate P _m that has not yet been a processing target step S8010 and subsequent The processing after S8010 is performed.

  The objective viewpoint index can be identified by the above processing.

  As described above, even with the information processing apparatus and the information processing method according to the present embodiment, the light emission timing control mechanism and the high-speed camera that are used in [Prior Art 3] are not required, and as a result, the cost is reduced. Can be kept low.

[Third Embodiment]
In the first embodiment, the objective viewpoint index is set in the imaging apparatus that moves in the space. The information processing apparatus according to the present embodiment is an information processing apparatus in which an objective viewpoint index is set on an arbitrary target object, and has a configuration in which the target object is added to the information processing apparatus of the first embodiment.

  FIG. 8 is a diagram illustrating a configuration of the information processing apparatus according to the present embodiment. As illustrated in FIG. 8, the information processing apparatus 900 according to the present embodiment includes an objective viewpoint camera 180, an image input unit 160, a data storage unit 170, a subjective viewpoint index detection unit 110, a position / orientation estimation unit 920, an objective viewpoint index. The candidate detection unit 140, the position constraint condition calculation unit 150, the objective viewpoint index identification unit 990, and the subjective viewpoint camera 930 are configured.

  Portions having the same functions as those in the first embodiment are denoted by the same reference numerals as those in FIG. However, this embodiment is different from the first embodiment in that an image acquired by the subjective viewpoint camera 930 fixed to the target object 935 is input to the image input unit 160 as a subjective viewpoint image.

  In the present embodiment, it is assumed that the objective viewpoint index is set on the target object 935.

  The subjective viewpoint camera 930 is fixedly attached to the target object 935. Here, it is assumed that the position and orientation of the target object 935 in the subjective viewpoint camera coordinate system are known.

  Further, it is assumed that an objective viewpoint index P whose position on the target object coordinate system is known is set on the target object 935 as an objective viewpoint index.

The position / orientation estimation unit 920 includes a set of the image coordinates u ^Qkn of each subjective viewpoint index detected by the subjective viewpoint index detection unit 110 and the world coordinates x _W ^Qkn previously stored as known information as the data storage unit 170. And the position and orientation of the subjective viewpoint camera 930 are estimated based on these pieces of information.

  Further, the position / orientation estimation unit 920 is based on the position and orientation of the subjective viewpoint camera 930 (in the world coordinate system) estimated as described above and the position and orientation of the target object 935 in the subjective viewpoint camera coordinate system that are known values. The position and orientation of the target object 935 are estimated.

  The objective viewpoint index identification unit 990 is based on the position and orientation of the target object 935 estimated by the position and orientation estimation unit 920 and the position constraint condition of each objective viewpoint index candidate calculated by the position constraint condition calculation unit 150. Identify objective perspective indicators.

  As described above, identification of the objective viewpoint index set for an arbitrary target object is realized.

  Note that the position / orientation estimation unit 920 in the present embodiment obtains the position and orientation of the target object 935 after once obtaining the position and orientation of the subjective viewpoint camera 930, but directly obtains the position and orientation of the target object 935. Also good.

Note that the target object 935 in the present embodiment may be an imaging device that captures a scene. Then, the subjective viewpoint camera 930 is arranged, for example, upward so as to have a field of view different from the field of view of the imaging device that captures a scene, and accordingly, the subjective viewpoint index Q _k is arranged in the field of view of the subjective viewpoint camera 930. Also good. By doing so, it becomes not enter subjective-view-indices Q _k in the field of view of an imaging apparatus for imaging a scene, to contribute to alleviate such a problem that impairs the appearance.

  In the present embodiment, by attaching a plurality of subjective viewpoint cameras 930 to the target object 935, the position and orientation of the target object 935 can be estimated with high accuracy in both position and orientation. Also good.

(Modification 1)
In each of the above-described embodiments, an inertial sensor may be attached to the imaging apparatus, and the position and orientation of the imaging apparatus may be estimated by, for example, the technique of Non-Patent Document 6. In this case, it is sufficient that at least two or more subjective viewpoint indices are always observed on the subjective viewpoint image acquired by the imaging device. In this case, the position and orientation of the imaging device can be stably estimated as compared with the case where only image information is used.

(Modification 2)
As the subjective viewpoint camera or objective viewpoint camera used in each of the above embodiments, a camera that captures light having a wavelength different from that of visible light may be used. For example, a camera that captures infrared light may be used as the objective viewpoint camera, and an index that emits or reflects infrared light may be used as the objective viewpoint index. In this case, since the subjective viewpoint index is not captured by the objective viewpoint camera, there is an effect that the subjective viewpoint index is not erroneously detected on the objective viewpoint image.

  It is also possible to use a camera that captures infrared light as the subjective viewpoint camera, and use an index that emits or reflects infrared light as the subjective viewpoint index. Furthermore, a camera that captures infrared light may be used for both the subjective viewpoint camera and the objective viewpoint camera, and an index that emits or reflects infrared light may be used for both the subjective viewpoint index and the objective viewpoint index.

  The camera that captures light having a wavelength different from that of visible light is not limited to a camera that captures infrared light, and a camera that captures ultraviolet light or the like may be used. Furthermore, you may use the camera which images simultaneously both the light of a wavelength different from visible light, and visible light.

(Modification 3)
The objective viewpoint index identification unit in each of the above embodiments has determined that index candidates whose distance from the estimated position of the objective viewpoint index is within a predetermined range are objective viewpoint indices. The distance from the estimated position may be obtained, and the smallest one may be determined as the objective viewpoint index. If the minimum distance is equal to or greater than the threshold, it may be determined that there is no index candidate corresponding to the objective viewpoint index.

(Modification 4)
In the objective viewpoint index identification unit in each of the above embodiments, the index is identified by obtaining the three-dimensional distance between the estimated position of the objective viewpoint index and the index candidate, but the two-dimensional distance on the objective viewpoint image is obtained. Thus, the index may be identified. That is, the position and orientation of the objective viewpoint camera in the world coordinates are input from the data storage unit 170, and the estimated position of the objective viewpoint index is projected on the objective viewpoint image based on these information, and the image coordinates and index candidate images are projected. The index may be identified by obtaining a two-dimensional distance from the coordinates. In this case, needless to say, the position constraint condition calculation unit is not necessary.

(Modification 5)
The objective viewpoint index identification unit in each of the above embodiments outputs the image coordinates of the objective viewpoint index to the outside via the I / F 1009. However, the information output to the outside is not limited to the image coordinates of the objective viewpoint index. For example, the position and orientation of the imaging device (or target object) may be calculated based on the image coordinates of the objective viewpoint index and the image coordinates of the subjective viewpoint index by the method of Patent Document 3, and output the same to the outside.

  FIG. 12 is a flowchart of processing for calculating the position and orientation of the imaging apparatus (or target object). The configuration for realizing this processing is a configuration in which a position and orientation calculation unit is added to the configuration of the first embodiment. For example, the CPU 1001 functions as a position and orientation calculation unit by executing a software program. In addition, the description about the same structure as other embodiment is abbreviate | omitted.

  In step S12000, the position / orientation calculation unit determines the image coordinates of the subjective viewpoint index detected by the subjective viewpoint index detection unit, the image coordinates of the objective viewpoint index identified by the objective viewpoint index identification unit, and the position constraint condition calculation unit. A parameter that constrains the position of the calculated objective viewpoint index is input. As a parameter for restricting the position of the objective viewpoint index, a parameter representing a three-dimensional line may be input when there is one objective viewpoint camera, and a three-dimensional position is input when there are two or more objective viewpoint cameras. do it.

  In step S12010, the position / orientation calculation unit determines the position of the imaging device (or target object) that minimizes the error of the subjective viewpoint index on the subjective viewpoint image under the constraint condition of the position where the objective viewpoint index should exist. And calculate the posture. Specifically, for example, the position and orientation of the imaging device (or target object) are calculated by the method of Patent Document 3.

  In step S12020, the position / orientation calculation unit outputs information on the calculated position and orientation of the imaging device (or target object) in the form of a modeling transformation matrix, for example.

  Rather than using parameters that constrain the position of the objective viewpoint index, an operation that minimizes the sum of errors between the image coordinates of the subjective viewpoint index and the objective viewpoint index and the calculated values of those points is performed. Thus, the position and orientation of the imaging device (or target object) may be calculated. Specifically, for example, the method of Patent Document 3 may be used.

(Modification 6)
In each of the above embodiments, first, the position and orientation of the imaging device (or target object) are estimated using the information of the subjective viewpoint image, and the objective viewpoint index is identified based on the position and orientation. However, the method of identifying the objective viewpoint index using the information of the subjective viewpoint image is not limited to this.

  For example, the objective viewpoint index may be identified by the method described below. First, it is assumed that each objective viewpoint index candidate is an objective viewpoint index. Next, using the image coordinates of the objective viewpoint index candidate and the subjective viewpoint index, which are assumed to be objective viewpoint indices, the position and orientation of the imaging apparatus are calculated by the method of Patent Document 3, for example. An objective viewpoint index candidate that minimizes the residual of the projection error obtained in the calculation process may be determined as the objective viewpoint index. In this case, it is not necessary to estimate the position and orientation of the imaging device using only the subjective viewpoint image. In this case, the position / orientation estimation unit and the position constraint condition calculation unit are not required.

(Modification 7)
In each of the above embodiments, the case where each subjective viewpoint index is identifiable by some method has been described. However, each subjective viewpoint index may not be identified in the same manner as the objective viewpoint index. That is, in FIG. 1, FIG. 5, or FIG. 8, the subjective viewpoint index Q _k (k = 1 to 4) may be configured by natural features such as circular markers having substantially the same color and edges. In this case, the objective viewpoint index can be identified by the method described below.

  FIG. 13 is a flowchart of processing for identifying an objective viewpoint index. This process functions as an objective viewpoint index identification unit, for example, when the CPU 1001 executes a software program.

  In step S13000, the objective viewpoint index identification unit inputs the image coordinates and three-dimensional position of each objective viewpoint index candidate, the image coordinates of each subjective viewpoint index candidate, and the three-dimensional position of each subjective viewpoint index from the data storage unit 170. To do.

  In step S13010, the objective viewpoint index identification unit temporarily sets one of the three-dimensional positions of the objective viewpoint index candidates as the position of the imaging device (or target object).

  In step S13020, the objective viewpoint index identification unit projects the three-dimensional position of each subjective viewpoint index onto a virtual sphere centered on the assumed position of the imaging device, and sets each of the subjective viewpoint index candidates, for example, as non-patent. Corresponding temporarily by the method of Reference 7.

  In step S13030, the objective viewpoint index identification unit uses the image coordinates of the objective viewpoint index candidate and the image coordinates of the subjective viewpoint index candidate temporarily associated in step S13020. Calculate the position and orientation. Then, the residual of the projection error obtained in the calculation process is obtained.

  In step S13040, the objective viewpoint index identification unit determines whether or not the processes of steps S13010 to S13030 have been performed on all objective viewpoint index candidates. If the process has not been completed for all objective viewpoint index candidates, the process returns to step S13010, and the processes of steps S13010 to S13030 are performed for unprocessed objective viewpoint index candidates. On the other hand, if the processing has been completed for all objective viewpoint index candidates, the process proceeds to step S13050.

  In step S13050, the objective viewpoint index identification unit compares the residual calculated in step S13030 for each objective viewpoint index candidate, determines that the objective viewpoint index candidate having the smallest objective viewpoint index is an objective viewpoint index, The data is output to the data storage unit 170.

  Note that the subjective viewpoint index information temporarily associated in step S13020 when the selected objective viewpoint index candidate is assumed may be simultaneously output as subjective viewpoint index identification information. Further, the position and orientation of the imaging device calculated in step S13030 when assuming the selected objective viewpoint index candidate may be output as the position and orientation of the imaging device.

  Through the above processing, the objective viewpoint index can be identified.

(Modification 8)
In the modified example 7, the case where the three-dimensional position of the objective viewpoint index candidate can be calculated, that is, the case where there are a plurality of objective viewpoint cameras, has been described, but there may be one objective camera. In this case, the objective viewpoint index can be identified by the method described below.

  FIG. 14 is a flowchart of processing for identifying an objective viewpoint index when there is one objective camera. This process functions as an objective viewpoint index identification unit, for example, when the CPU 1001 executes a software program.

  In step S14000, the objective viewpoint index identification unit inputs the image coordinates of each objective viewpoint index candidate, the image coordinates of each subjective viewpoint index candidate, and the three-dimensional position of each subjective viewpoint index from the data storage unit 170.

  In step S14010, the objective viewpoint index identification unit selects one objective viewpoint index candidate.

  In step S14020, the objective viewpoint index identification unit divides the three-dimensional straight line that constrains the position of the selected objective viewpoint index, for example, at an interval of 10 cm, and temporarily sets each of the divided positions as the position of the imaging device.

  In step S14030, the objective viewpoint index identification unit projects the three-dimensional position of each subjective viewpoint index onto a virtual sphere centered on the assumed position of the imaging apparatus, and sets each of the subjective viewpoint index candidates, for example, as non-patent. Corresponding temporarily by the method of Reference 7.

  In step S14040, the objective viewpoint index identification unit uses, for example, the method of Patent Document 3 using the image coordinates of the selected objective viewpoint index candidate and the image coordinates of the subjective viewpoint index candidate temporarily associated in step S14030. The position and orientation of the imaging device are calculated. Then, the residual of the projection error obtained in the calculation process is obtained.

  In step S14050, the objective viewpoint index identification unit determines whether or not the processing in steps S14020 to S14040 has been performed on all assumed positions of the imaging apparatus. If the process has not been completed for all assumed positions, the process returns to step S14020, and the processes of steps S14020 to S14040 are performed for unprocessed assumed positions. On the other hand, if the process has been completed for all assumed positions, the process proceeds to step S14060.

  In step S14060, the objective viewpoint index identification unit compares the residuals calculated at all assumed positions of the imaging apparatus and selects the minimum value.

  In step S14070, the objective viewpoint index identification unit determines whether or not the processes of steps S14010 to S14060 have been performed on all objective viewpoint index candidates. If the process has not been completed for all objective viewpoint index candidates, the process returns to step S14010, and the processes of steps S14010 to S14060 are performed for unprocessed objective viewpoint index candidates. On the other hand, if the processing has been completed for all objective viewpoint index candidates, the process proceeds to step S14080.

  In step S14080, the objective viewpoint index identification unit compares the minimum value of the residual selected in step S14060 for each objective viewpoint index candidate, and the objective viewpoint index candidate having the smallest value is the objective viewpoint index. Determine and output to the data storage unit 170.

  As in the case of the modification example 7, it is of course possible to output the identification information of the subjective viewpoint index obtained when assuming the selected objective viewpoint index candidate and the position and orientation of the imaging device.

  Through the above processing, the objective viewpoint index can be identified.

(Modification 9)
The method of associating the subjective viewpoint index and the subjective viewpoint index candidate in Step S13020 of Modification 7 and Step S14030 of Modification 8 is not limited to the method of Non-Patent Document 7. For example, by attaching an inertial sensor to the imaging apparatus and measuring the absolute value of the tilt angle and making only the azimuth angle unknown, more efficient association can be performed.

  FIG. 15 is a flowchart of processing for identifying a subjective viewpoint index. This process is an alternative process to Step S13020 of Modification Example 7 and Step S14030 of Modification Example 8. For example, the CPU 1001 functions as an objective viewpoint index identification unit by executing a software program. Here, it is assumed that the measured value of the tilt angle is input from the data storage unit 170 in advance.

  In step S15000, the objective viewpoint index identification unit tentatively sets an unknown azimuth angle, for example, in increments of 1 degree.

  In step S15010, the objective viewpoint index identification unit projects the three-dimensional position of each subjective viewpoint index on the captured image based on the tilt angle, the assumed azimuth angle, and the assumed position of the imaging apparatus. Then, each of the subjective viewpoint indices is associated with each of the subjective viewpoint index candidates based on the two-dimensional distance between the projection coordinates of each subjective viewpoint index and the image coordinates of the subjective viewpoint index candidates. For example, each subjective viewpoint index is temporarily associated with a subjective viewpoint index candidate having an image coordinate closest to the projected coordinate.

  In step S15020, the objective viewpoint index identification unit calculates an evaluation value for associating the subjective viewpoint index and the subjective viewpoint index candidate in step S15010. For example, the two-dimensional distance between the projected coordinates of each subjective viewpoint index and the image coordinates of the subjective viewpoint index candidates associated with each other is obtained, and the number of associations such that the distance becomes smaller than a preset threshold is determined. An evaluation value may be used. Or you may use the evaluation value which considered both each two-dimensional distance and the said number. For example, the sum of the number obtained by subtracting the mean square value of the two-dimensional distance from the threshold value and the number may be used as the evaluation value. Of course, the present invention is not limited to these, and any evaluation value can be used as long as it can measure the degree of correctness of association.

  In step S15030, the objective viewpoint index identification unit determines whether or not the processing in steps S15000 to S15020 has been performed for all assumed azimuth angles. If the process has not been completed for all assumed azimuth angles, the process returns to step S15000, and the processes of steps S15000 to S15020 are performed for unprocessed assumed azimuth angles. On the other hand, if the process has been completed for all assumed azimuth angles, the process proceeds to step S15040.

  In step S15040, the objective viewpoint index identification unit compares the evaluation values calculated in step S15020 for all assumed azimuth angles. Then, a correlation result between the subjective viewpoint index and the subjective viewpoint index candidate that maximizes the evaluation value is selected.

  Thus, by making the inclination angle known, the time required for the process of associating the subjective viewpoint index with the subjective viewpoint index candidate can be shortened compared to the method of Non-Patent Document 7.

  When an inertial sensor is attached to the imaging device, the association between the subjective viewpoint index and the subjective viewpoint index candidate is performed based on a one-dimensional search. Therefore, even when a natural feature such as an edge is used for the subjective viewpoint index, it can be associated with higher accuracy than before.

  Here, for example, the azimuth angle search range may be narrowed down to a compass measurement error range (for example, ± 30 degrees) by mounting a magnetic compass on the imaging apparatus and measuring a rough azimuth angle.

  Using the objective viewpoint index and the subjective viewpoint index identified as described above, the position and orientation of the imaging apparatus can be calculated by the method of Patent Document 3, for example.

  Note that even when there is only one objective viewpoint index candidate or when the objective viewpoint index can be identified by some other means, that is, when the objective viewpoint index has already been identified, The identification method of the viewpoint index is effective.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows schematic structure of the information processing apparatus which concerns on 1st Embodiment. And FIG. 11 is a diagram illustrating a basic configuration of a computer that can realize each unit of the information processing apparatus by software. It is a flowchart explaining the process which calculates the straight line which restrains the position of the objective viewpoint parameter | index candidate in 1st Embodiment. It is a flowchart explaining the process which identifies the objective viewpoint parameter | index in 1st Embodiment. It is a figure which shows schematic structure of the information processing apparatus which concerns on 2nd Embodiment. It is a flowchart explaining the process which calculates the three-dimensional position of the objective viewpoint index candidate in 2nd Embodiment. It is a flowchart explaining the process which identifies the objective viewpoint parameter | index in 2nd Embodiment. It is a figure which shows schematic structure of the information processing apparatus which concerns on 3rd Embodiment. It is an example of the parameter | index currently utilized with the prior art. It is a figure which shows the example of the real space where a light-emitting body exists other than a parameter | index. It is a figure which shows the image imaged with the infrared camera shown in FIG. 16 is a flowchart for describing processing for calculating the position and orientation of an imaging apparatus (or target object) in Modification 5. 18 is a flowchart for describing processing for identifying an objective viewpoint index in Modification 7. It is a flowchart explaining the process which identifies the objective viewpoint parameter | index in the modification 8. 22 is a flowchart for describing processing for identifying a subjective viewpoint index in Modification 9.

Claims (7)

A first image input step of inputting a first image captured by an imaging device that captures a scene;
A second image input step of inputting a plurality of second images imaged by a stereo objective viewpoint imaging means from a plurality of objective viewpoint positions fixed to the imaging device;
A tilt angle input step of inputting a tilt angle measurement value from a tilt angle measuring device that measures information related to the tilt angle of the imaging device;
A temporary setting step of temporarily setting a plurality of azimuth angles of the imaging device;
A first detection step of detecting a feature amount of a first index in the scene from the first image input in the first image input step;
A second detection step of detecting a feature quantity of a candidate for a second index included in the imaging device from each of a plurality of second images input in the second image input step;
A first index position calculating step of calculating a plurality of projection positions of the first index for each of the plurality of azimuth angles set temporarily;
Based on the feature amount of the first index detected in the first detection step, by selecting one of the projection positions of the plurality of first indicators, the detected feature amount The first index is identified as indicating the first index of the selected projection position, and the tentatively set azimuth corresponding to the calculation of the selected projection position is used as the azimuth angle of the imaging apparatus. A first identification step;
A second index position calculation step of calculating a three-dimensional position of the second index based on the feature amount detected in the first detection step;
A second candidate position calculating step of calculating a three-dimensional position of the second index candidate based on the feature amount detected from each of the plurality of second images in the second detection step;
A second identification step of identifying whether or not the candidate indicates the second index based on a three-dimensional distance between the three-dimensional position of the second index and the three-dimensional position of the candidate. An information processing method characterized by the above.   The information processing method according to claim 1, wherein the second index is provided in the imaging device via the object provided in the object provided in the imaging device. The information processing method according to claim 1 , wherein the processing by the first identification step is executed along with the processing of the first detection step. The information processing method according to claim 1 , wherein the processing by the first identification step is executed along with the processing of the second identification step. The computer program for making a computer perform the information processing method of any one of Claims 1 thru | or 4 . A computer-readable recording medium storing the computer program according to claim 5 . First image input means for inputting a first image picked up by an image pickup device for picking up a scene;
Second image input means for inputting a plurality of second images picked up by a stereo objective viewpoint image pickup means from a plurality of objective viewpoint positions fixed to the image pickup device;
A tilt angle input means for inputting a tilt angle measurement value from a tilt angle measuring device that measures information related to the tilt angle of the imaging device;
Provisional setting means for provisionally setting a plurality of azimuth angles of the imaging device;
First detection means for detecting a feature quantity of the first index in the scene from the first image input by the first image input means;
Second detection means for detecting a feature quantity of a candidate for a second index included in the imaging device from each of a plurality of second images input by the second image input means;
First index position calculating means for calculating a plurality of projection positions of the first index for each of the plurality of temporarily set azimuth angles;
Based on the feature quantity of the first index detected by the first detection means, by selecting any one of the projected positions of the plurality of first indices, the detected feature quantity The first index is identified as indicating the first index of the selected projection position, and the tentatively set azimuth corresponding to the calculation of the selected projection position is used as the azimuth angle of the imaging apparatus. A first identification means;
Second index position calculation means for calculating a three-dimensional position of the second index based on the feature amount detected by the first detection means;
Second candidate position calculating means for calculating a three-dimensional position of the candidate for the second index based on a feature amount detected from each of the plurality of second images by the second detecting means;
Second identification means for identifying whether or not the candidate indicates the second index based on a three-dimensional distance between the three-dimensional position of the second index and the three-dimensional position of the candidate. An information processing apparatus characterized by the above.

Images