JP2009033366A - Optical marker system - Google Patents

Abstract

An optical marker system capable of stably estimating the position and orientation of a moving photographing means without using a special photographing means.
LED markers (21 to 2m) are attached at predetermined positions and blink with a unique blinking pattern uniquely assigned to the LED markers (21 to 2m). At this time, the camera 30 captures an image including the light emission points and feature points of the LED markers 21 to 2m, and the candidate point detection unit 40 detects the light emission points and feature points included in the captured image as candidate points. The light emission state detection unit 50 tracks each candidate point to detect a blinking pattern, the marker specifying unit 60 specifies the LED markers 21 to 2m from the blinking pattern, and the camera position / posture estimation unit 70 includes a marker specifying unit. The position and orientation of the camera 30 are estimated from the three-dimensional positions of the LED markers 21 to 2m identified by the reference numeral 60.
[Selection] Figure 1

Description

  The present invention relates to an optical marker system that detects an optical marker that transmits identification information.

  For people with higher brain dysfunction, even in familiar places such as home, the current location and the route to the destination may not be known. Therefore, a caregiver (family) always needs to be accompanied by a small amount of movement, and the burden on the caregiver is large. Therefore, in recent years, wearables that estimate the user's position and posture for the purpose of memory assistance and movement support in indoor environments such as when a person with higher brain dysfunction goes to a toilet and returns Research on systems is ongoing.

Various methods have been proposed so far for estimating the user's position and orientation. For example, a marker ID (identification information) is acquired while capturing an image using a high-speed image sensor. (Refer to Non-Patent Document 1), and taking a marker arranged in a square using a normal visible light camera to search for the shape of the marker arrangement (see Non-Patent Document 2).
Nobuyuki Matsushita et al., “Id cam: Smart camera capable of acquiring scene and id simultaneously”, linguistic theory, Vol. 43, no. 12, 2001, pp. 3664-pp. 3674 Tsuyoshi Aoki, “Infrared tags read by cameras and their applications”, Interactive System and Software VIII, 2000, pp. 131-pp. 136

  However, in the former method, it is necessary to use a dedicated image sensor, and it cannot be applied to an image shot with a general camera. In the latter method, since the marker arrangement shape is detected, the number of markers that can be arranged in the scene is limited, and the method cannot be applied when the position and orientation of the camera change.

  The objective of this invention is providing the optical marker system which can track accurately the optical marker contained in the image image | photographed with the imaging | photography means to move.

  Another object of the present invention is to provide an optical marker system that can stably estimate the position and orientation of a moving photographing means without using a special photographing means.

  An optical marker system according to the present invention includes a plurality of optical markers that are attached at predetermined positions and transmit identification information uniquely assigned to the optical markers by changing a light emission state, An imaging unit that captures an image including a light emitting point and a feature point that is fixed in the imaging space and has a predetermined brightness, and a candidate point that has a predetermined brightness is detected from each image captured by the imaging unit Each candidate point is tracked by comparing the candidate points detected by the candidate point detecting means and the candidate points detected by the candidate point detecting means between the images and associating the candidate points between the images. A light emission state detection unit that detects a light emission state and a specifying unit that specifies the optical marker from a plurality of candidate points based on the light emission state detected by the light emission state detection unit.

  In the optical marker system according to the present invention, a plurality of optical markers attached at predetermined positions transmits identification information uniquely assigned to itself by changing its light emission state, An image including a light emitting point of the optical marker and a feature point that is fixed in the imaging space and has a predetermined brightness is captured by the imaging means that moves in the imaging space. Then, a plurality of candidate points having a predetermined brightness are detected from each image photographed by the photographing means, and each candidate point is correlated between the images by comparing the detected candidate points between the images. Then, each candidate point is tracked to detect the light emission state of each candidate point, and an optical marker is specified from the plurality of candidate points based on the detected light emission state.

  Thus, not only the light emission point of the optical marker but also a feature point that is fixed in the imaging space and has a predetermined brightness is detected as a candidate point, so even if the optical marker is turned off, A point can be detected stably. Therefore, by associating candidate points between images using feature points that are stably detected, it is possible to stably track an optical marker that is in a light-off state or in a low light level in a certain image. Thus, the identification information of the optical marker can be detected stably. As a result, the position of the optical marker can be identified from the identification information of the optical marker, and the position and orientation of the moving imaging means can be stably estimated from the identified optical marker position without using a special imaging means. it can.

  Further, it is preferable that the specifying unit specifies a candidate point having a predetermined blinking pattern among the plurality of candidate points as the optical marker.

  In this case, a candidate point having a predetermined blinking pattern among the plurality of candidate points is specified as the optical marker, so that the optical marker can be reliably extracted from the plurality of candidate points, and the optical marker can be stably Can be tracked.

  Further, the plurality of optical markers are blinked so that a ratio of a light emission period to a light extinction period is approximately 0.5 and a blinking pattern is different from each other, and the specifying unit includes a light emission period among a plurality of candidate points. It is preferable that a candidate point having a ratio between the turn-off period and the turn-off period of about 0.5 is specified as the optical marker, and identification information of the optical marker is specified from the blinking pattern of the candidate point specified as the optical marker.

  In this case, among the plurality of candidate points, a candidate point having a ratio of the light emission period to the light extinction period of about 0.5 is identified as the light emission point of the optical marker, and the optical marker is identified from the blinking pattern of the identified candidate point. Since the information is specified, the light emission point of the optical marker can be reliably extracted from the plurality of candidate points, and the identification information of the optical marker can be reliably detected.

  The light emission state detection unit compares each candidate point detected by the candidate point detection unit between images to calculate a geometric relationship between images, and uses the calculated geometric relationship to next to each candidate point of the current frame. Predicts the observation position in the frame and associates the candidate point of the next frame located within the specified range from the predicted observation position with the candidate point of the current frame, thereby tracking each candidate point and detecting the blink pattern of each candidate point It is preferable to do.

  In this case, each candidate point is compared between the images to calculate a geometric relationship between the images, and using this geometric relationship, an observation position in the next frame of each candidate point of the current frame is predicted, and the predetermined position is determined from this observation position. Since the candidate point of the next frame located in the range is associated with the candidate point of the current frame, each candidate point can be accurately tracked.

  The specifying unit specifies identification information of the optical marker from a blinking pattern of candidate points specified as the optical marker, specifies a position of the optical marker on an image captured by the imaging unit, and the optical marker The system stores in advance a three-dimensional position of the optical marker in the imaging space in association with the identification information of each optical marker, and refers to the storage unit to determine the optical marker specified by the specifying unit. A position detection unit that identifies a three-dimensional position of the optical marker from the identification information and detects the position and orientation of the photographing unit from the three-dimensional position of the identified optical marker and the position on the image identified by the identification unit; It is preferable to further provide.

  In this case, the identification information of the optical marker is specified from the blinking pattern of the candidate points specified as the optical marker, the position of the optical marker on the photographed image is specified, and further, from the identification information of the optical marker Since the three-dimensional position of the optical marker is specified, and the position and posture of the photographing means are detected from the three-dimensional position of the specified optical marker and the position on the image, the position and posture of the photographing means can be accurately detected. It is possible to accurately estimate the position and posture of the user wearing this photographing means.

  The plurality of optical markers preferably transmit identification information by turning on or off infrared light.

  In this case, since the optical marker emits infrared light that cannot be directly seen by humans, it is possible to prevent the blinking pattern from being a visual interference with human behavior in the scene.

  According to the present invention, not only the light emission point of the optical marker but also a feature point that is fixed in the shooting space and has a predetermined brightness is detected as a candidate point, so even when the optical marker is turned off. The feature points can be stably detected, and the candidate points are associated with each other using the feature points that are stably detected. Since the optical marker in a low state can be stably tracked, the identification information of the optical marker can be stably detected, and the position of the optical marker is identified from the identification information of the optical marker. The position and orientation of the moving photographing means can be stably estimated without using the photographing means.

  Hereinafter, an optical marker system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an optical marker system according to an embodiment of the present invention. The optical marker system shown in FIG. 1 includes a lighting control device 10, m (m is an integer of 2 or more) LED markers 21 to 2m, a camera 30, a candidate point detection unit 40, a light emission state detection unit 50, and a marker specification unit 60. The camera position / posture estimation unit 70 and the marker position storage unit 80 are provided.

  The LED markers 21 to 2m are configured by LEDs (Light Emitting Diodes) that emit infrared light, a drive circuit, and the like, and are attached to a predetermined position in the shooting scene, for example, on the same plane. The three-dimensional position is stored in advance in the marker position storage unit 80 in association with the identification information of the LED markers 21 to 2m. Moreover, each LED marker 21-2m light-emits the blink pattern from which a blink pattern differs mutually, and the ratio of a light emission period and a light extinction period becomes substantially 0.5, and each of the LED markers 21-2m by this blink pattern Identification information uniquely assigned to the identification information is specified.

  The lighting control device 10 is composed of a computer or the like that executes a predetermined blinking control program, and blinks the LED markers 21 to 2m so as to transmit identification information uniquely assigned to each of the LED markers 21 to 2m. Control the state individually. In addition, the structure of the lighting control apparatus 10 is not specifically limited to said example, You may comprise a part or all from a dedicated hardware circuit, and it is individually comprised in the LED markers 21-2m. And the lighting control device 10 may be omitted.

  The light emitted from the LED markers 21 to 2m by the blinking control of the lighting control device 10 is preferably infrared light. In this case, since infrared light is not directly visible to humans, it is not a visual obstacle to human behavior in the scene. Further, light in a shorter wavelength region (wavelength region closer to visible light) in the infrared region is more preferable, and infrared light having a wavelength of 890 nm is used in this embodiment. In this case, since the normal visible light video camera has sufficient detection sensitivity in the wavelength range, the LED markers 21 to 2m emit light while shooting the scene using the normal visible light video camera. Light can be received with high accuracy.

  In addition, as light which LED marker 21-2m light-emits, it is not specifically limited to said example, You may use the light of visible region. In this case, the light emitted from the LED markers 21 to 2m can be photographed using a wavelength region in which a normal video camera for photographing visible light has sufficient sensitivity.

  The camera 30 is composed of a normal visible light video camera (for example, MV-GS200 manufactured by Panasonic) having sufficient sensitivity to infrared light. The camera 30 is mounted on a user, for example, the user's head so that the shooting direction coincides with the user's line-of-sight direction, the light emitting point of any of the LED markers 21 to 2m, fixed in the shooting space, and a predetermined amount. An image including a feature point having brightness, for example, a furnishing item in a shooting space including a texture having a luminance component (or color component) equal to or greater than a predetermined value, a user's usage, and the like is shot.

  The configuration of the camera 30 is not particularly limited to the above example, and various modifications are possible. For example, the LED markers 21 to 2m are photographed by the infrared light camera, or the LED markers 21 to 2m are visible. A visible light camera having sufficient sensitivity only to visible light when emitting light may be used. The camera 30 may be attached to a person other than a human user.

  FIG. 2 is a diagram illustrating an example of a blinking pattern of LED markers. In FIG. 2, for ease of illustration, four LED markers M1 to M4 among the LED markers 21 to 2m are illustrated. Yes. In FIG. 2, the high level indicates “ON” that is the lighting period, and the low level indicates “OFF” that is the extinguishing period, and the LED markers M <b> 1 to M <b> 4 are basically synchronized with the frame period of the camera 30. Repeats blinking.

  As shown in FIG. 2, the LED marker M1 is “ON”, “OFF”, “ON”, “ON”, “OFF”, “ON”, “OFF”, “OFF” over the first to eighth frames. The blinking pattern changes in synchronization with the frame period, and this blinking pattern is used as a basic pattern, and blinks in the same manner after 9 frames. The LED marker M2 has a blinking pattern that changes from “ON”, “ON”, “OFF”, “ON”, “OFF”, “OFF”, “ON”, “OFF” over the first to eighth frames. The blinking pattern is used as a basic pattern, and blinks in the same manner after 9 frames. The LED marker M3 has a blinking pattern of “ON”, “ON”, “OFF”, and “OFF” in the first to fourth frames. The blinking pattern is used as a basic pattern and blinks in the same manner for the fifth and subsequent frames. To do. The blinking pattern of the LED marker M4 changes from “ON” to “OFF” in the first and second frames. The blinking pattern is used as a basic pattern and blinks in the third and subsequent frames as well.

  When the above blinking pattern “ON” is encoded as “1” and “OFF” is encoded as “0”, the LED markers M1 to M4 are 8 of “10110100”, “11010010”, “11001100”, and “10101010”, respectively. Identification information having a bit pattern of bits is transmitted. Since each of these identification information has a different bit pattern and the number of “1” and “0” is equal, the LED markers M1 to M4 transmit uniquely assigned identification information, And it blinks with the blink pattern whose ratio of a lighting period and a light extinction period is set to 0.5. The blinking states of the LED markers other than the four LED markers M1 to M4 are basically the same as described above except that the blinking patterns are different from each other.

  FIG. 3 is a diagram illustrating an example of an image captured by the camera 30. The example shown in FIG. 3 is an example in which a predetermined image is printed on the main surface of the box, four LED markers M1 to M4 are attached to predetermined positions on the main surface, and the main surface of the box is photographed. An image including the light emitting points m1 to m4 of the four LED markers M1 to M4 and a plurality of feature points Q indicated by white circles (high luminance points in the printed image) is taken by the camera 30, and a plurality of The feature point Q includes pixels that are detected as candidate points in the same manner as the light emission points of the LED markers 21 to 2m when the candidate point detection unit 40 described later performs template matching. The light emission points m1 to m4 and the plurality of feature points Q of the markers M1 to M4 are detected as candidate points.

  FIG. 4 is a diagram schematically showing an outline of the optical marker system shown in FIG. As shown in FIG. 4, a plurality of LED markers M1 to M4 are installed in the imaging space, and each LED marker M1 to M4 blinks with a unique blinking pattern as described above. Then, the light emission points of the LED markers M1 to M4 are tracked from the image sequence of the scene including the light emission points and feature points of the LED markers M1 to M4 photographed by the camera 30 worn by the user by a marker specifying process or the like described later. From the blinking pattern of each light emitting point, it is specified which LED marker M1 to M4 indicates each light emitting point, and the camera 30 using a predetermined three-dimensional position of the specified LED marker M1 to M4. Are estimated.

  Again referring to FIG. 1, candidate point detection unit 40, light emission state detection unit 50, marker identification unit 60, camera position and orientation estimation unit 70, and marker position storage unit 80 are configured integrally with camera 30 and are wearable computers. Is attached to the user. The configuration of each block such as the candidate point detection unit 40 is not particularly limited to the above example, and a part or all of the blocks may be configured by a dedicated hardware circuit. In addition, the candidate point detection unit 40 and the like are configured separately from the camera 30, and image data captured by the camera 30 is transmitted to the candidate point detection unit 40 and the like by wireless communication to perform the processing after the candidate point detection unit 40. Or the image data captured by the camera 30 is transmitted to a predetermined server via a network such as the Internet, and the processing after the candidate point detection unit 40 is performed online on the server, or the captured image is offline from the captured image. Processing may be performed.

The candidate point detection unit 40 performs template matching using a predetermined template for each image taken by the camera 30, and detects a plurality of candidate points having a predetermined brightness. Here, the LED markers 21~2m and feature points of lighting conditions, sufficiently bright in the input image, the candidate point detecting unit 40, the luminance component V _t of the observed image I _t at time t, by performing the horizontal templates T _h for detecting the horizontal edges, the template matching using a vertical template T _v for detecting the vertical edge, the light-emitting point and the characteristic point of the LED marker 21~2m Are detected as candidate points and output to the light emission state detection unit 50.

Specifically, the light emitting state detection unit 50, the luminance component V _t of the image I _t, 2-dimensional position where each score is above a certain value which is calculated by the equation (1) and (2) below (X, y) is extracted as a candidate point. Here, in Expressions (1) and (2), ε is a threshold value of a pixel that can be determined as the light emitting point of the LED markers 21 to 2m. Further, m is a horizontal template T _h, and shows the horizontal size of the vertical template T _v, n denotes a horizontal template T _h, and the vertical size of the vertical template T _v.

Figure 5 is a diagram showing an example of a template used in the candidate point detecting unit 40 shown in FIG. 1 shows a (a) shows a horizontal template T _h, (b) vertical template T _v. The candidate point detection unit 40 scans each image using the horizontal template _Th and the score J _h according to the equation (1) is larger than ε, and scans each image using the vertical template T _v to obtain the equation ( score J _v by 2) a larger pixel epsilon, detected as candidate points.

  Referring to FIG. 1 again, the light emission state detection unit 50 compares each candidate point detected by the candidate point detection unit 40 between images, and associates each candidate point between images, thereby The candidate points are tracked to detect the blinking pattern of each candidate point and output to the marker specifying unit 60.

  Specifically, the light emission state detection unit 50 compares each candidate point detected by the candidate point detection unit 40 between images, calculates a geometric relationship between images, for example, planar projective transformation, and calculates the calculated geometric relationship. For example, the observation position in the next frame of each candidate point of the current frame is predicted using planar projective transformation, and the candidate point of the next frame located within a predetermined range from the predicted observation position is associated with the candidate point of the current frame. Thus, each candidate point is tracked to detect a blinking pattern of each candidate point.

  In addition, if there is no candidate point within a predetermined range from the predicted observation position in the next frame, the light emission state detection unit 50 determines that the candidate point is turned off in the next frame, and is turned off. From the positional relationship between the determined candidate point and another candidate point in the current frame and the positional relationship of the other candidate point in the next frame, the position in the next frame of the candidate point determined to be unlit is calculated.

  The marker identifying unit 60 identifies the light emitting points of the LED markers 21 to 2m from among the plurality of candidate points based on the blinking pattern detected by the light emitting state detecting unit 50, and the blinking patterns of the identified LED markers 21 to 2m. Identification information of the LED markers 21 to 2m is specified, and the positions of the LED markers 21 to 2m on the captured image are specified and output to the camera position and orientation estimation unit 70.

  Specifically, the marker specifying unit 60 specifies candidate points having a ratio of the light emission period to the light extinction period of about 0.5 among the plurality of candidate points as the light emission points of the LED markers 21 to 2m, and other candidates. Identify points as feature points or noise. And the marker specific | specification part 60 encoded the blink pattern of the candidate point specified as the light emission point of LED marker 21-2m with "1", "0", and was uniquely allocated with respect to LED marker 21-2m. Among the blinking patterns (identification information), the blinking pattern (identification information) corresponding to the obtained code string (flashing pattern) (high correlation) is detected as the blinking pattern (identification information) of the LED markers 21 to 2m. Moreover, the marker specific | specification part 60 detects the two-dimensional position of the LED markers 21-2m on the image image | photographed with the camera 30. FIG.

  The marker position storage unit 80 stores the three-dimensional positions of the LED markers 21 to 2m in advance in association with the identification information of the LED markers 21 to 2m. The camera position / orientation estimation unit 70 acquires the identification information of the LED markers 21 to 2m specified by the marker specifying unit 60, and stores the three-dimensional positions of the LED markers 21 to 2m stored in association with the acquired identification information. The three-dimensional position and orientation of the camera 30 are estimated based on the three-dimensional position read from the marker position storage unit 80 and the two-dimensional positions of the LED markers 21 to 2m specified by the marker specifying unit 60.

  In the present embodiment, the LED markers 21 to 2m correspond to an example of an optical marker, the camera 30 corresponds to an example of an imaging unit, the candidate point detection unit 40 corresponds to an example of a candidate point detection unit, and a light emission state detection The unit 50 corresponds to an example of the light emission state detection unit, the marker specification unit 60 corresponds to an example of the specification unit, the camera position / posture estimation unit 70 corresponds to an example of the position detection unit, and the marker position storage unit 80 stores the storage unit It corresponds to an example.

  Next, camera position and orientation estimation processing by the optical marker system shown in FIG. 1 will be described. FIG. 6 is a flowchart for explaining camera position and orientation estimation processing by the optical marker system shown in FIG.

First, in step S11, the camera 30 captures a plurality of images including any of the LED markers 21 to 2m in a state where the LED markers 21 to 2m are blinking according to a unique blinking pattern. Get the image sequence. Next, in step S12, the candidate point detecting section 40, to the image sequence, for detecting a candidate point having a predetermined brightness performs template matching using a horizontal template T _h and vertical template T _v.

  Next, in step S13, the light emission state detection unit 50 obtains a planar projective transformation H between the two preceding and following images in the image sequence acquired in step S1, and calculates the relative relationship between the two preceding and following images. Specifically, the light emission state detection unit 50 calculates the planar projective transformation H as follows.

In the images I ^(t) and I ^{(t + 1)} observed at times t and t + 1, n _t and n _{t + 1} candidate points p _i ^(t) (i = 1,..., N _t ) and p _j ^{( t + 1)} (j = 1,..., n _{t + 1} ) is obtained. Here, assuming that the candidate points are distributed on a certain plane, the two-dimensional observation position on the images I ^(t) and I ^{(t + 1)} is represented by p _k ^(t) = [x _{k for} the three-dimensional point P _k on the plane. ^{Assuming that (t)} , y _k ^(t) ], p _k ^{(t + 1)} = [x _k ^{(t + 1)} , y _k ^{(t + 1)} ], the following relationship is obtained.

H shown in Expression (3) is a plane projective transformation, and indicates a relationship between the images I ^(t) and I ^{(t + 1)} . Of the candidate point groups p _i ^(t) (i = 1,..., N _t ), p _j ^{(t + 1)} (j = 1,..., N _{t + 1} ) in the images I ^(t) and I ^{(t + 1)} , p _i For a set of candidate points having a correspondence relationship between ^(t) and p _j ^{(t + 1)} , the same relationship as in equation (3) is obtained, and thus the planar projective transformation H is calculated as follows.

First, provisional association is performed for p _i ^(t) and p _j ^{(t + 1)} based on the distance on the image. Based on the set of candidate points obtained here, for example, a set of candidate points corresponding to the planar projective transformation H that best satisfies Equation (3) using RANSAC (Random Sample Consensus), p _k ^{(t _{), p k (t + 1}} ) (k = k 1, ..., by estimating the _{k m),} calculate the homography H. Here, as a temporary association between p _k ^(t) and p _k ^{(t + 1)} , for example, among p _j ^{(t + 1)} , the one having the shortest distance to p _k ^(t) is p _k ^{(t + 1).} It may be specified as

Next, in step S ^<b> 14, the light emission state detection unit 50 tracks each candidate point included in the images I ^(t) and I ^{(t + 1)} using the plane projection transformation H described above. Specifically, the light emission state detection unit 50 tracks each candidate point as follows.

First, consider the i-th candidate point p _i ^(t) obtained at time t. When p _i ^(t) is converted by the planar projective transformation H, a predicted observation position p ′ _i ^{(t + 1)} = Hp _i ^(t) of p _i ^(t) at time t + 1 is obtained. Here, if the distance between the two points p ′ _i ^{(t + 1)} and p _j ^{(t + 1)} is represented by d (p ′ _i ^{(t + 1)} and p _j ^{(t + 1)} ), the actually detected candidate point p _j ^{(t + 1) )} (J = 1,..., N _{t + 1} ) If there is a candidate point p _j ^{(t + 1)} that satisfies the following expression (4), the LED marker of the candidate point p _i ^(t) is lit at time t + 1. The candidate point is tracked by determining that it exists at the two-dimensional position p _j ^{(t + 1 )} and associating p _j ^{(t + 1)} with p _i ^(t) .

d (p ′ _i ^{(t + 1)} , p _j ^{(t + 1)} ) <ε _d (4)
In the case where the candidate point that satisfies the equation (4) there are a ^{_{plurality, d (p 'i (t}} + 1), p j (t + 1)) a distance of a candidate point that minimizes _p ^{j (t + 1)} a _p ^{i ( t)} . Further, as ε _d , a predetermined threshold that can be considered that p _j ^{(t + 1)} and p _i ^(t) are candidate points of the same LED marker can be adopted.

On the other hand, if there is no candidate point that satisfies Expression (4), it is determined that the LED marker of p _i ^(t) is turned off at time t + 1. In this case, for example, from the positional relationship between the feature points other than _p ^{i (t)} and _p ^{i (t)} at time _t, determine the position at time t + 1 of ^{p i (t),} the candidate point the position at time t + 1 What is necessary is just to match | combine with p _i ^(t) as p _i ^{(t + 1)} . That is, if a part of noise is removed, the light emitting point by the LED markers 21 to 2m and the feature point move in the same manner as the position and orientation of the camera 30 change, and thus p _i ^{(t) at} time t. And the light emission points of the LED markers 21 to 2m that are extinguished at the time t + 1 can be calculated from the positional relationship between the feature points other than p _i ^(t) and the light emission points can be tracked without interruption.

In addition, when the tracking of the candidate point proceeds to some extent and the light emitting state detection unit 50 knows which LED marker 21 to 2m indicates the light emitting point of a certain candidate point of the current frame, the LED marker 21 to 2m Based on the blinking pattern, candidate points for the next frame may be specified. For example, when there are a plurality of candidate points for the next frame that satisfy Expression (4), the distance is closest to the predicted observation position p ′ _i ^{(t + 1)} and the time t + 1 from the blinking pattern of the LED marker at p _i ^(t) The candidate point having the same blinking state as the predicted blinking state may be specified as the candidate point of the next frame of p _i ^(t) .

  Next, in step S15, the light emission state detection unit 50 detects the blink pattern of each candidate point from the tracking result of each candidate point obtained in step S4. Here, for example, the light emission state detection unit 50 indicates the blinking state of each candidate point at a certain candidate point by “ON” at time t, “OFF” at time t + 1, and “ON” at time t + 2. The blinking pattern of each candidate point is detected by arranging in series.

  Next, in step S <b> 16, the marker specifying unit 60 specifies whether each candidate point is a light emitting point, a feature point, or noise of the LED marker from the blinking pattern of each candidate point detected in step S <b> 15. . Here, candidate points other than the light emitting points of the LED markers 21 to 2m include feature points and noise that is neither a light emitting point nor a feature point, but the blinking pattern of the candidate points based on the feature points is almost “ON”. As for the candidate points due to noise, the blinking pattern is almost “OFF”. The LED markers 21 to 2m arranged in the imaging space are set so that the ratio of “ON” to “OFF” is substantially “1: 1”.

  Therefore, when the ratio of “ON” to “OFF” of a certain candidate point is approximately “1: 1”, the marker specifying unit 60 specifies this candidate point as the light emitting point of the LED markers 21 to 2m, and a certain candidate When the ratio of the point “ON” to “OFF” is approximately “1: 0”, this candidate point is specified as a feature point, other candidate points are specified as noise, and the LED markers 21 to 2 m emit light. Extract points separately from others. As described above, the marker specifying unit 60 can accurately distinguish the feature points and noise from the light emitting points of the LED markers 21 to 2m by obtaining the ratio of the candidate points “ON” and “OFF”. .

  Next, in step S <b> 17, the marker specifying unit 60 specifies the two-dimensional position on the captured image of the candidate points specified as the light emitting points of the LED markers 21 to 2 m. In addition, the marker specifying unit 60 encodes the blink pattern of the candidate points specified as the light emitting points of the LED markers 21 to 2m with “1” and “0”, and the obtained code string corresponds to the LED markers 21 to 2m. By identifying which identification information among the uniquely assigned identification information is identified, which light emitting point of each of the LED markers 21 to 2m is identified, the identification information of the LED marker Is detected. For example, in the example of FIG. 2, when the blinking pattern of a certain candidate point is “10110100”, the marker specifying unit 60 matches the identification information of the LED marker M1, so this candidate point is used as the light emission point of the LED marker M1. And “10110100” is specified as the identification information of the LED marker M1.

  Next, in step S <b> 18, the camera position / orientation estimation unit 70 3 of LED markers 21 to 2 m associated with the identification information of the LED markers 21 to 2 m specified by the marker specifying unit 60 from the marker position storage unit 80. The dimensional position is read out, and the position and orientation of the camera 30 in each frame are estimated based on the read out three-dimensional position and the two-dimensional positions of the LED markers 21 to 2m specified by the marker specifying unit 60, and then The processing after step S11 is continued, and the position and orientation of the camera 30 are sequentially estimated.

Here, the camera position and orientation estimation unit 70 estimates the position and orientation of the camera 30 as follows. The three-dimensional coordinates of three-dimensional points P _i (i = 1,..., N) distributed on the same plane in the three-dimensional position space are photographed by the camera 30 with X _i = [X _i , Y _i , Z _i ] ^t . When the two-dimensional projection position in image and m _i, it is possible to obtain the relationship of equation (5) below by planar projective transformation.

m _i = aHX _i (5)
However, m _i = [x _i , y _i , f] ^t , f indicates the focal position of the camera 30, and a indicates a constant. Further, the planar projective transformation H is defined by the following equation (6). Here, T, R is the position of the camera 30 in the X _i coordinate system indicates the attitude, d represents the distance between the between the camera 30 and the plane, n represents shows the normal vector of the plane.

H = dR + Tn ^t (6)
Accordingly, when four or more candidate points are obtained, the camera position / orientation estimation unit 70 can determine the plane projection transformation H by Equation (5), and decomposes the plane projection transformation H by Equation (6). By doing so, the position and orientation of the camera 30 can be detected.

  With the above processing, in this embodiment, each LED marker 21 to 2m blinks in a blinking pattern uniquely assigned to itself, so that the candidates observed in the image photographed by the camera 30 are displayed. By detecting the blinking pattern of the dots, it is possible to specify which LED marker 21 to 2m the light emitting point is indicated by the observed candidate point. Here, in addition to the light emitting points of the LED markers 21 to 2m, feature points having a predetermined brightness existing in the imaging space are also tracked as candidate points, so that the LED markers 21 to 2m in the unlit state are displayed. Also in the captured image, candidate points for the LED markers 21 to 2m can be specified, and the LED markers 21 to 2m can be tracked with high accuracy. Further, since the LED markers 21 to 2m are attached at predetermined positions, it is possible to estimate the position and orientation of the user wearing the camera 30 without using special photographing means.

  Next, the optical marker system according to the present embodiment will be described with a specific example. FIG. 7 is a schematic diagram illustrating an arrangement example of four LED markers. As shown in FIG. 7, after four LED markers M1 to M4 are arranged around a rectangular area of 70 mm × 50 mm, the camera 30 is moved and rotated while the scene including the four LED markers M1 to M4 is moved. A video sequence was photographed and the above processing was performed on the photographed image.

  FIG. 8 is a diagram showing the measurement results of the trajectories of the four LED markers M1 to M4 extracted from the captured image sequence, and FIG. 9 is the measurement result of the blinking pattern of the four LED markers M1 to M4. FIG. From the results shown in FIGS. 8 and 9, the trajectories of the four LED markers M1 to M4 could be stably obtained, and the blinking patterns of the LED markers M1 to M4 could be detected almost accurately. Note that the blinking pattern of the LED marker M2 cannot be acquired between the frames 80 to 120. This is because the LED marker was not correctly detected by template matching.

  The position and posture of the camera obtained by the present invention can be used for various purposes. For example, the current position of a person with higher brain dysfunction is estimated at home, or stored in movement in an indoor environment. The present invention can be applied to a wearable system such as assistance or movement support.

It is a block diagram which shows the structure of the optical marker system by one embodiment of this invention. It is a figure which shows an example of the blink pattern of the LED marker shown in FIG. It is the figure which showed an example of the image which the camera shown in FIG. 1 image | photographed. It is the figure which showed typically the outline | summary of the optical marker system shown in FIG. It is a figure which shows an example of the template used in the candidate point detection part shown in FIG. It is a flowchart for demonstrating the camera position and orientation estimation process by the optical marker system shown in FIG. It is a schematic diagram which shows the example of arrangement | positioning of an LED marker. It is a figure which shows the measurement result of the locus | trajectory of four LED markers. It is a figure which shows the measurement result of the blink pattern of four LED markers.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Lighting control apparatus 21-2m LED marker 30 Camera 40 Candidate point detection part 50 Light emission state detection part 60 Marker specific part 70 Camera position and orientation estimation part 80 Marker position memory | storage part

Claims (6)

A plurality of optical markers attached to predetermined positions and transmitting identification information uniquely assigned to itself by changing a light emission state;
Photographing means for photographing an image including a light emitting point of the optical marker and a feature point fixed in a photographing space and having a predetermined brightness;
Candidate point detection means for detecting candidate points having a predetermined brightness from each image photographed by the photographing means;
A light emitting state in which each candidate point detected by the candidate point detecting means is compared between images, and each candidate point is correlated between images to track each candidate point and detect a light emitting state of each candidate point Detection means;
An optical marker system comprising: specifying means for specifying the optical marker from among a plurality of candidate points based on the light emission state detected by the light emission state detection means.   The optical marker system according to claim 1, wherein the specifying unit specifies a candidate point having a predetermined blinking pattern among the plurality of candidate points as the optical marker. The plurality of optical markers are blinked so that a ratio of a light emission period and a light extinction period is approximately 0.5 and blink patterns are different from each other.
The specifying means specifies, as the optical marker, a candidate point having a ratio of a light emission period to a turn-off period of about 0.5 among a plurality of candidate points, and uses the blinking pattern of the candidate point specified as the optical marker to The optical marker system according to claim 2, wherein identification information of the marker is specified.   The light emission state detection unit compares each candidate point detected by the candidate point detection unit between images to calculate a geometric relationship between images, and uses the calculated geometric relationship to next to each candidate point of the current frame. Predicts the observation position in the frame and associates the candidate point of the next frame located within the specified range from the predicted observation position with the candidate point of the current frame, thereby tracking each candidate point and detecting the blink pattern of each candidate point The optical marker system according to any one of claims 1 to 3, wherein: The specifying unit specifies identification information of the optical marker from a blinking pattern of candidate points specified as the optical marker, specifies a position of the optical marker on an image shot by the shooting unit,
Storage means for storing in advance the three-dimensional position of the optical marker in the imaging space in association with the identification information of each optical marker;
Referring to the storage means, the three-dimensional position of the optical marker is specified from the identification information of the optical marker specified by the specifying means, and the three-dimensional position of the specified optical marker and the image specified by the specifying means 5. The optical marker system according to claim 1, further comprising a position detection unit that detects a position and an attitude of the photographing unit from the position.   The optical marker system according to claim 1, wherein the plurality of optical markers transmit identification information by turning on or off infrared light.