JP6982865B2 - Moving image distance calculation device and moving image distance calculation program - Google Patents

Description

The present invention relates to a moving image distance calculation device and a moving image distance calculation program, and more specifically, it is reflected in a moving image based on a moving image obtained by taking a front view in the traveling direction with a moving camera. The present invention relates to a moving image distance calculation device for calculating the distance from an object to a camera and a moving image distance calculation program.

In recent years, a camera for photographing the outside world in the front direction is often installed on a moving object such as a vehicle or a drone. Recently, there is a demand to use the captured moving image not only to capture the front view with a camera but also to acquire the distance information necessary for automatic driving of vehicles and drones.

As a method of measuring the distance between a moving object and another object, a method of installing a distance sensor or the like on the moving object is often used. As the distance sensor, for example, a method of directly measuring the distance to another object by using a laser sensor, an ultrasonic sensor, an infrared sensor or the like is used (see, for example, Non-Patent Document 1).

Recently, a method of calculating the distance to another object taken by image analysis of a moving image taken by a stereo camera has also been used (see, for example, Non-Patent Document 2).

Naoki Suganuma, "Cognitive / Judgment Technology in Autonomous Autonomous Vehicles-From On-Board Sensing to Path Planning-", Proceedings of the 23rd Image Sensing Symposium (SSII2017), Yokohama (Pacifico Yokohama), June 2017 Shuhei Takimoto, Takaaki Ito, "Development of Monocular Distance Measurement Verification System Using In-Vehicle Camera", SEI Technical Review, No. 169, p.82-87, July 2006

Generally, when a distance sensor is installed in a vehicle, a plurality of distance sensors are often installed in one vehicle. For example, ultrasonic sensors are installed to detect other objects that are relatively close to the vehicle. A laser sensor is installed in order to detect other objects that are relatively far from the vehicle. In order to improve the accuracy of information by the distance sensor, it is necessary to install a plurality of distance sensors of several types, which causes a problem of cost increase. In particular, since the laser sensor is more expensive than other sensors, there is a problem that the cost burden increases.

In addition, although the laser sensor can measure the distance to an object in front, it identifies and identifies the measurement target such as whether the object is a person, a vehicle, or an obstacle. There was a problem that it was difficult.

When measuring the distance to the object in front using a stereo camera, it is possible to identify and specify the object to be measured by image analysis processing or the like. However, in the case of a stereo camera, at least two or more cameras are required, which causes a problem of cost increase.

Furthermore, when using a stereo camera, it is necessary to determine which object in the outside world the point cloud of the measured distance corresponds to. Therefore, even when using a stereo camera, a distance sensor is often installed.

In addition, using multiple distance sensors to determine the distance to the object in front leads to an increase in the cost for sensor installation as a whole, and is complicated to integrate the data acquired by those sensor groups. There was a problem that various processing was required. A similar problem has occurred when measuring the distance to another object not only in a vehicle but also in a moving object such as a drone.

The present invention has been made in view of the above problems, and the distance from the object reflected in the moving image to the camera is determined based on the moving image in which the front view in the traveling direction is taken by one moving camera. An object of the present invention is to provide a moving image distance calculation device for calculation and a moving image distance calculation program.

In order to solve the above problem, the moving image distance calculation device according to the present invention uses one moving camera to change the front view in the moving direction with time t = τ−T (however, τ> 0, T> 0). It is a moving image distance calculation device that calculates the distance from the object reflected in the moving image to the camera based on the moving image taken from the time t = τ, and the time t = τ−T of the moving image. A target pixel extraction means for extracting one of the pixels of an object reflected in the frame image at the time as a target pixel, and extracting the target pixel for each of M (M ≧ 2) different objects from the frame image, and time t = Based on the plurality of frame images from τ−T to time t = τ, the trajectory of the coordinates of the M target pixels extracted by the target pixel extraction means is dynamically planned for the two-dimensional image. Based on the method, the locus calculation means for calculating from the frame image at time t = τ −T to the frame image at time t = τ in chronological order, and the M loci calculated by the locus calculation means. based on the time t = tau-T the time from the coordinates of the frame image t = number of mobile pixels up coordinates of the frame image of tau _q m of (m = 1,2, ···, M ) a, M pieces wherein to the number of pixels moved measuring means for measuring for each target pixel, among the measured M-number of the mobile number of pixels q _m by the moving pixel number measuring means, and the number of pixels the number of moving pixels is smallest μ of, _Let γ be the number of pixels having the largest number of moving pixels, and let Zn be the shortest distance from the object specified by the M target pixels to the camera, and the M target pixels be used. The farthest distance from the specified object to the camera is Z _L , and the constant a and the constant b are set to Z L.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (γ-μ)) log (Z _L / Z _N )
The distance Z at the time t = τ from the object corresponding to each of the M target pixels to the camera is defined as Z _m (m = 1, 2, ..., M). _m is based on the constant a and the constant b and the number of M moving pixels q _m .
It is characterized by having a distance calculation means calculated by Z _m = a · exp (−bq _m).

Further, in the moving image distance calculation program according to the present invention, the front view in the moving direction is changed from time t = τ−T (where τ> 0, T> 0) to time t = τ by one moving camera. It is a moving image distance calculation program for calculating the distance from the object reflected in the moving image to the camera based on the moving images taken up to, and the time t = τ− of the moving image is used as a control means. A target pixel extraction function that uses one of the pixels of the object displayed in the frame image at the time of T as the target pixel and extracts the target pixel for each of M (M ≧ 2) different objects from the frame image, and time. Based on the plurality of frame images from t = τ−T to time t = τ, the locus of the coordinates of the M target pixels extracted by the target pixel extraction function is the motion for the two-dimensional image. A locus calculation function for calculating in chronological order from the frame image at time t = τ −T to the frame image at time t = τ based on the target planning method, and M of the above calculated by the locus calculation function. _{Based on the locus, the number of moving pixels q m} (m = 1, 2, ..., M) from the coordinates of the frame image at time t = τ −T to the coordinates of the frame image at time t = τ is determined. and M of the measured to the number of pixels moved measurement function for each target pixel, among the measured M-number of the mobile number of pixels q _m by the number of pixels moved measurement function, the number of pixels the number of moving pixels is smallest μ _Let γ be the number of pixels having the largest number of moving pixels, and let Zn be the shortest distance from the object specified by the M target pixels to the camera, and M the targets. _{Let Z L be} the farthest distance from the object specified by each pixel to the camera, and let constant a and constant b be.
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
Calculated by b = (1 / (γ-μ)) log (Z _L / Z _N ), the distance from the object corresponding to each of the M target pixels to the camera at time t = τ is Z. _{As m} (m = 1, 2, ..., M), the distance Z _m is set based on the constant a and the constant b and the number of M moving pixels q _m .
It is characterized by realizing a distance calculation function calculated by Z _m = a · exp (−bq _m).

Further, by applying the mean-shift method to the frame image used for extracting the target pixel in the moving image distance calculation device or the moving image distance calculation program described above, a plurality of the frame images can be obtained. It is divided into regions, and among the divided regions, the pixel at the center of gravity of the region where the object is reflected is used as the target pixel, or the pixels in the region are numbered in order from the end to be the region. The pixel corresponding to the intermediate number among the pixel numbers of the above may be used as the target pixel, and the target pixel may be extracted for each of M (M ≧ 2) regions in which different objects are reflected.

According to the moving image distance calculation device and the moving image distance calculation program according to the present invention, from the object reflected in the moving image to the camera based on the moving image in which the front view in the traveling direction is taken by the moving camera. It becomes possible to calculate the distance.

It is a block diagram which showed the schematic structure of the moving image distance calculation apparatus which concerns on embodiment. It is a figure which showed the state in the front direction seen from the moving object schematically. It is a figure for demonstrating the data of a moving image and a target pixel. It is a figure for demonstrating the tracking process of the target pixel which changes with the lapse of time. It is a figure for demonstrating the method of finding the virtual distance to an object based on cumulative dynamic parallax. It is a figure which showed the pixel group which can transition from the frame image at the time t-1 to one target pixel (x, y, t) at the time t. It is an enlarged view of the frame image which showed the locus when the target pixel which consists of the pixel of the area where a traffic light is reflected is tracked from time t = 1 to time t = 30. It is a frame image which showed the locus when the target pixel of 5 traffic lights was tracked from time t = 1 to time t = 30, respectively. It is a figure which showed the image which applied the mean-shift method to the frame image taken by the camera. It is a block diagram which showed the functional part of the function which is executed by the CPU according to a program. (A) is a method in which a CPU according to an embodiment calculates a distance from a camera to an object by performing tracking processing of target pixels in chronological order from time t = τ−T to time t = τ. It is a flowchart which showed. In (b), the CPU according to the embodiment determines the distance from the camera to the object by performing tracking processing of the target pixels in chronological order retroactively from time t = τ to time t = τ−T. It is a flowchart which showed the calculation method.

Hereinafter, an example of the moving image distance calculation device according to the present invention will be shown and described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a moving image distance calculation device. The moving image distance calculation device 100 includes a recording unit 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a CPU (Central Processing Unit: control means, area division means, target pixel extraction means, distance). It has a calculation means, a locus calculation means, a moving pixel number measuring means, a distance calculation means) 104.

A camera 200 is connected to the moving image distance calculation device 100. The camera 200 is mounted on, for example, a vehicle or a drone. The camera 200 can capture a moving image of a moving vehicle or the like in front of the vehicle in the traveling direction. Further, a monitor 210 is connected to the moving image distance calculation device 100. The monitor 210 can display a moving image or the like taken by the camera 200.

A moving image taken by the camera 200 is recorded in the recording unit 101. More specifically, a moving image is recorded in the recording unit 101 as data obtained by recording a plurality of frame images in time series. For example, consider the case where a moving image from time t = 1 to time t = T is taken by the camera 200. If one frame image (frame image) can be recorded as a moving image of the camera 200 every Δt time, the T / Δt frame images are recorded in the recording unit 101 in chronological order. Become. As an example, the recording unit 101 according to the present embodiment can record T frame images for each unit time.

A frame buffer is provided in the moving image distance calculation device 100 or the camera 200, and the frame image for each unit time taken by the camera 200 is temporarily recorded in the frame buffer, and the frame image recorded in the frame buffer is recorded. , It may be configured to be recorded in the recording unit 101 in chronological order. Further, the moving image recorded in the recording unit 101 is not limited to the moving image taken in real time by the camera 200, and may be a moving image (past moving image) taken by the camera 200 in advance.

Further, the moving image taken by the camera 200 is not limited to the digital moving image. For example, even if the captured moving image is an analog moving image, if it is possible to record the frame image in the recording unit 101 in time series by the digital conversion process, the distance of the moving image distance calculation device 100 It can be used for calculation processing.

The recording unit 101 is composed of a general hard disk or the like. The configuration of the recording unit 101 is not limited to the hard disk, and may be a flash memory, SSD (Solid State Drive / Solid State Disk), or the like. The specific configuration of the recording unit 101 is not particularly limited as long as it is a recording medium capable of recording a moving image as a plurality of time-series frame images.

Based on a plurality of frame images (moving images) recorded in time series in the recording unit 101, the CPU 104 sets the distance from the object displayed in the frame image to the camera 200 to the pixels in which the object is displayed. Perform the calculation process accordingly. The CPU 104 performs a distance calculation process for each specific pixel (target pixel) according to a processing program (a program based on the flowchart of FIGS. 11A and 11B) described later, and the details thereof will be described later.

A program or the like is recorded in the ROM 102 to calculate the distance from the object shown in the frame image to the camera 200. The RAM 103 is used as a work area used for processing the CPU 104.

In the moving image distance calculation device 100 according to the present embodiment, a case where a program (flow chart shown in FIGS. 11A and 11B) executed by the CPU 104 is recorded in the ROM 102 will be described. However, these programs may be recorded in the recording unit 101.

The camera 200 is a photographing means capable of photographing a landscape in front of the camera as a moving image through a lens. The type and configuration of the camera 200 is not particularly limited as long as it is possible to capture a moving image. For example, it may be a general movie camera, or it may be one that uses a camera function of a smartphone or the like.

The monitor 210 provides the user with a moving image taken by the camera 200, an image showing the locus of pixels tracked by the distance calculation process (for example, images of FIGS. 7 and 8 described later). It is possible to display it visually. A general display device such as a liquid crystal display or a CRT display is used for the monitor 210.

Next, a concept of calculating the distance of an object reflected in the frame image based on the frame images recorded in the recording unit 101 in chronological order will be described.

More than 2000 years ago, Euclid discussed the visual phenomenon of motion parallax. The visual phenomenon due to dynamic parallax is a phenomenon in which when an object is moving at a constant velocity, the movement of a distant object is visually smaller than that of a nearby object. FIG. 2 is a diagram schematically showing a state in the front direction as seen from a moving object. As shown in FIG. 2, when the object moves forward, the object located in the center (see A in FIG. 2) does not move much. Observing the surroundings other than the center (see B in FIG. 2), the movement of distant objects is small, and the movement of nearby objects is large. Visual phenomena due to such dynamic parallax are observed on a daily basis. This visual phenomenon is the subject of psychological research. In addition, technology that applies this visual phenomenon is being used for animation processing using computer graphics and for developing simulators for pilot driving technology education.

The moving image distance calculation device 100 utilizes a visual phenomenon due to dynamic parallax to obtain a camera 200 from an object reflected in the moving image from a moving image taken by a camera mounted on a moving vehicle, a drone, or the like. Calculate the distance to.

FIG. 3 is a diagram for explaining the moving image data and the target pixel. The moving image taken by the camera 200 is represented by f (x, y, t). The range of the variable x is 1 ≦ x ≦ X, the range of the variable y is 1 ≦ y ≦ Y, and the variable t is t = 1, 2, ..., T. X indicates the number of horizontal pixels of the moving image captured by the camera 200, and Y indicates the number of vertical pixels. f (x, y, t) is the value of the corresponding pixel, and usually indicates the value of RGB (R: red, G: green, B: blue). (X, y) indicates the coordinate point of the corresponding pixel, and t indicates the time (relative time).

By indicating the moving image by f (x, y, t), the moving image data can be represented as rectangular parallelepiped data as shown in FIG. The foremost surface of FIG. 3 shows a frame image f (x, y, t) at time t = 1 (however, 1 ≦ x ≦ X, 1 ≦ y ≦ Y). The pixel f (x ₀ , y ₀ , 1) at any of the coordinates (x ₀ , y ₀ ) in this frame image is set as the target pixel. In FIG. 3, for convenience of explanation, the case where there is one target pixel will be described, but the number of target pixels is not limited to one.

Consider tracking how a target pixel moves in a rectangular parallelepiped over time t to reach which pixel in the frame image at time t = T. By considering the tracking of the target pixel, how the object recorded in the moving image looks from the viewpoint of the visual phenomenon of dynamic parallax when the front view is taken by a moving camera. It is possible to detect whether it changes.

FIG. 4 is a diagram for explaining a tracking process of a target pixel that changes with the passage of time. In FIG. 4, the target pixel f (x ₀ , y _{0 , 1) at the coordinates (x 0} , y ₀ ) at the time t = 1 and the coordinates (x ^* ₀ (T)) at the time t = T are shown. , Y ^* ₀ (T)) and the target pixel f (x ^* ₀ (T), y ^* ₀ (T), T). Tracking the target pixel is called tracking. The target pixel is located in the frame image at time t = 1 from the position (x ₀ , y ₀ , 1) (= (x ₀ , y ₀ )) in the frame image at time t = T (x ^*. Consider the case of moving to ₀ (T), y ^* _{0 (T)).} At this time, if the cumulative dynamic parallax of the target pixel due to the elapsed time T from the time t = 1 to the time t = T is q (x ₀ , y ₀ , T), the cumulative dynamic parallax q (x ₀ , y). ₀ , T) is
q (x ₀ , y ₀ , T) = | (x ₀ , y ₀ )-(x ^* ₀ (T), y ^* ₀ (T)) |
Can be determined as.

The cumulative dynamic parallax q (x ₀ , y _{0 , T) is reflected in the coordinates (x 0} , y ₀ ) (= f (x ₀ , y ₀ , 1) coordinates of the frame image at time t = 1). The object position (object point) in the real world of the object is reflected in ^{the coordinates (x *} ₀ (T), y ^* ₀ (T)) on the frame image at time t = T. It corresponds to the change in the distance from the camera 200 to the object position (object point) in the real world when moving to (object point). By finding the correspondence between the amount of change in the cumulative dynamic parallax and the amount of change in the distance from the camera 200 to the object position in the real world, from the camera 200 to the object position in the real world based on the cumulative dynamic parallax q. Distance can be calculated.

FIG. 5 is a diagram for explaining a method of obtaining a distance from an object to the camera 200 based on cumulative dynamic parallax. The vertical axis of FIG. 5 shows the virtual distance Zv from the camera 200 to the object. The positive direction of the virtual distance Zv is the lower direction in the figure. The horizontal axis of FIG. 5 indicates the cumulative dynamic parallax q. The positive direction of the cumulative dynamic parallax q is the right direction in the figure. Assuming that the camera 200 advances in the front direction, the virtual distance is shortened by ΔZv (> 0) and the cumulative dynamic parallax is changed by Δq (> 0 or <0) with the passage of time of Δt. Since the value of the virtual distance Zv is virtual, it corresponds to the value _{of the unit cumulative dynamic parallax q 0, which is a constant.} As a characteristic of dynamic parallax, there is a phenomenon that the larger the value of dynamic parallax, the shorter the distance from the camera 200 to the object, and the smaller the value of dynamic parallax, the longer the distance from the camera 200 to the object. Therefore, a proportional relationship is established between the parameters of the virtual distance Zv and the cumulative dynamic parallax q _0. That is, from the relationship shown in FIG. 5, _{the proportional relationship of Zv: q 0} = −ΔZv: Δq is established.

From this proportional relation, _{the relational expression of −q 0} · ΔZv = Zv · Δq is established.
ΔZv / Zv = −Δq / q ₀
By transforming the equation with logZv = -q / q ₀ + c (c is a constant)
Zv = a · exp (-bq)
Is established. Here, a and b are positive constants. Further, exp (-bq) indicates the value of the base of the natural logarithm (Napier's constant) to the power of -bq. When the values of the constants a and b are determined, the value of Zv is obtained as the distance in the real world instead of the virtual distance by calculating the value of the cumulative dynamic parallax q based on the moving image taken by the camera 200. Will be possible.

The values of the constants a and b are determined based on the fluctuation range of the variable Zv and the variable q. Zv indicates the virtual distance from the camera 200 to the object position, as described above. The virtual distance is a value that can change depending on the target world (target world, target environment). For example, the target world of Zv in the present embodiment is a virtual three-dimensional space of a moving image, which depends on the fluctuation of the cumulative dynamic parallax value of the target pixel. The virtual distance obtained in the three-dimensional space (target world) of the moving image is a value different from the distance in the real world. Therefore, by observing or otherwise determining in advance the fluctuation range of the distance in the real world corresponding to the virtual distance Zv in the three-dimensional space (target world) of the moving image, the distance in the target world can be determined in advance to the real world. It is possible to find the distances in association with each other.

If the distance Z in the real world can be associated with the virtual distance Zv in the target world,
Z = a · exp (−bq) ・ ・ ・ Equation 1
Allows the distance Z in the real world to be obtained. That is, it becomes possible to obtain the distance Z from the camera 200 to the object position in the real world.

In the present embodiment, as an example, the fluctuation range of the distance in the real world corresponding to the virtual distance Zv in the three-dimensional space (target world) of the moving image is determined by inspection. That is, it is determined by visual observation of the user or the like how much the virtual distance Zv in the three-dimensional space (target world) of the moving image is included in the real world.

For example, when calculating the distances of M objects reflected in a frame image, M virtual distances Zv exist in the three-dimensional space (object world) of the moving image. The real-world distance range corresponding to the M virtual distances Zv (the real-world distance range in which it can be determined that all the distances from the camera 200 to each of the M objects are included, and is visually determined. The distance range in the real world) is defined as the distance range from Z _{N to Z L} (Z _N ≤ Z _L ). The distance range of the virtual distance Zv can be expressed _{by Z N} ≤ Zv ≤ Z _{L. The distance Z N} and the distance Z _L indicating the distance range of the virtual distance Zv are not necessarily limited to those determined visually, and may be determined by other methods.

The fluctuation range of the value of the cumulative dynamic parallax q is determined by the experimental value individually obtained from the moving image. That is, no a priori information is required. The fluctuation range of the cumulative dynamic parallax q is a fluctuation range that depends on the real world of the object shown in the moving image and also depends on the time range T. Here, the cumulative dynamic parallax q can be obtained by how far the target pixel in the frame image at time t = 1 has moved in the frame image at time t = T. That is, the cumulative dynamic parallax q can be obtained by the number of moving pixels of the target pixel moved from the coordinates of the frame image at time t = 1 to the coordinates of the frame image at time t = T. For example, when there are M target pixels corresponding to each of the M objects, the cumulative dynamic parallax (number of moving pixels) can also be obtained as M according to the number of target pixels. The fluctuation range of the cumulative dynamic parallax (number of moving pixels) q of M obtained in this way is set to μ ≦ q ≦ γ. The smallest number of pixels among the M moving pixels corresponds to μ, and the largest number of M moving pixels corresponds to γ. That is, μ and γ are experimental values determined by the number of moving pixels of the plurality of target pixels obtained by the moving image.

The correspondence between μ, γ and Z _L , Z _N can be determined based on the nature of dynamic parallax. μ _{corresponds to Z L} and γ corresponds to Z _N. This is because the farther the virtual distance Zv is, the smaller the amount of movement of the object point (object position) of the moving image is, and the closer the virtual distance Zv is, the larger the amount of movement of the object point (object position) of the moving image is. This is due to the nature of dynamic parallax. _{In this way, the distance Z N} having the shortest distance in the distance range of the virtual distance Zv corresponds to γ having the largest amount of movement in the fluctuation range of the cumulative dynamic parallax (number of moving pixels) q, and the virtual distance Zv. _{The distance Z L} , which has the longest distance in the distance range, corresponds to μ, which has the smallest amount of movement in the fluctuation range of the cumulative dynamic parallax (number of moving pixels) q.

Therefore, by substituting the values of Zv and q of Zv = a · exp (−bq) with μ and Z _L , and γ and Z _N , the following simultaneous equations for a and b are established. ..

Z _L = a · exp (−bμ) ・ ・ ・ Equation 2
Z _N = a · exp (−bγ) ・ ・ ・ Equation 3
By solving this simultaneous equation, the constants a and b can be obtained as follows.

a = Z _L · exp ((μ / (γ −μ)) log (Z _L / Z _N )) ・ ・ ・ Equation 4
b = (1 / (γ-μ)) log (Z _L / Z _N ) ・ ・ ・ Equation 5
In this way, by obtaining the constants a and b and applying them to the above-mentioned equation 1, the value of the virtual distance Zv can be calculated as the distance Z in the real world.

Next, the tracking method of the target pixel will be described. FIG. 6 shows a pixel group (local region) that can transition from a frame image at time t-1 to one target pixel (x, y, t) at time t, that is, a frame image that can be a transition target. It is a figure which showed the pixel group. When tracking pixels at time t using dynamic programming for a two-dimensional image, as shown in FIG. 6, any pixel in the local region at time t-1 to be tracked. Will be transitioned to the target pixel at time t. In this respect, it can be determined that FIG. 6 shows the local transition in the dynamic programming method for the two-dimensional image to be pixel-tracked. Based on this local transition, a recurrence formula of dynamic programming for a two-dimensional image is created.

Here, the dynamic programming method for a two-dimensional image is a method for obtaining a non-linear correspondence between two series, and is one application of a generally known dynamic programming method. Applies to the example. The difference from the conventional dynamic programming method is how to make the local distance. In the dynamic programming method for a two-dimensional image used in the present embodiment, the local distance is calculated in the relationship between the pixel value of the target pixel and the value of all the pixels of the moving image. Then, a path that gives the minimum cumulative distance from the start point to the end point is obtained by using dynamic programming, with the target pixel as the start point and all the pixels of the image at time t = T as possible end points. By using such a technique, it becomes possible to track the target pixel over a plurality of frame images from time t = 1 to time t = T.

The recurrence formula of the dynamic programming method for a two-dimensional image first uses a target pixel f (x ₀ , y ₀ , 1) and a value called a local distance, d (x, y, t) = | Calculate f (x ₀ , y ₀ , 1) -f (x, y, t) |. The boundary condition of d (x, y, t) is d (x, y, for t ≦ 0, or x ≦ 0, or x> X, or y ≧ 0, or y> Y. t) = ∞. Further, the value obtained by accumulating the optimum local distance is defined as the optimum cumulative distance S (x, y, t). The initial condition of the optimum cumulative distance S (x, y, t) is set to 0 ≦ x ≦ X + 1, 0 ≦ y ≦ Y + 1, 0 ≦ t ≦ T, and S (x ₀ , y ₀ , 1) = 0. , S (x, y, t) = ∞.

・・・式６
となる。 When such boundary conditions and initial conditions are set, the recurrence formula of the optimum cumulative distance based on dynamic programming is used.

... Equation 6
Will be.

Here, it is defined as N (i, j) ≡ {(i + α, j + β): | α | ≤α ₀ , | β | ≤β _0}. N (i, j) shows a set of grid points of the frame image at time t-1 shown in FIG. 6, and shows the range in which the recurrence formula by dynamic programming is involved.

・・・式７
ここで、記号ａｒｇは、最小値を与えることになる変数ｘ，ｙを取り出すという操作を意味している。このとき、ターゲットピクセル（ｘ_０，ｙ_０）の時間Ｔによる累積動的視差をｑ（ｘ_０，ｙ_０，Ｔ）とすると、累積動的視差ｑ（ｘ_０，ｙ_０，Ｔ）は、
ｑ（ｘ_０，ｙ_０，Ｔ）＝｜（ｘ_０，ｙ_０）−（ｘ^＊ _０（Ｔ），ｙ^＊ _０（Ｔ））｜
と定めることができる。この累積動的視差ｑ（ｘ_０，ｙ_０，Ｔ）の関係式を用いて、ターゲットピクセルを追跡することにより、ターゲットピクセルの軌跡（トラッキング）に基づいて、カメラ２００から対象物までの距離を求めることができる。 When the recurrence formula shown in Equation 6 is applied from t = 1 to t = T, the optimum cumulative value is the distribution of scalar values of S (x, y, T) (however, 1 ≦ x ≦ X, 1). ≦ y ≦ Y). The optimum cumulative value S (x, y, T) when acted from t = 1 to t = T is a value equal to or less than a preset threshold value h, and the coordinates are obtained to be the minimum value. Assuming that these coordinates are (x ^* ₀ (T), y ^* ₀ (T)), these coordinates can be obtained by the following equation.

・ ・ ・ Equation 7
Here, the symbol arg means an operation of extracting variables x and y that give a minimum value. At this time, if the _{cumulative dynamic parallax of the target pixel (x 0} , y ₀ ) due to the time T is q (x ₀ , y ₀ , T), the cumulative dynamic parallax q (x ₀ , y ₀ , T) is.
q (x ₀ , y ₀ , T) = | (x ₀ , y ₀ )-(x ^* ₀ (T), y ^* ₀ (T)) |
Can be determined. By tracking the target pixel using the relational expression of the cumulative dynamic parallax q (x ₀ , y ₀ , T), the distance from the camera 200 to the object is determined based on the trajectory (tracking) of the target pixel. You can ask.

Here, the trajectory of the target pixel in the moving image can be determined by the recurrence formula described below. The locus of the target pixel in the moving image is important information for visualizing the change of the target pixel.

To find the locus of the target pixel, from the destination (x ^* ₀ (T), y ^* ₀ (T), T) to (x ^* ₀ (1), y ^* ₀ (1), 1) = (x) _The sequence of points up to 0, y ₀ , 1) is obtained by a recurrence formula.

First, the target pixel of the coordinates (x ^* ₀ (T-1), y ^* ₀ (T-1)) is set.
(X ^* ₀ (T-1), y ^* ₀ (T-1))
= (X ^* ₀ (T) + i ^* , y ^* ₀ (T) + j ^* ).

によって定まる。 In this case, (i ^* , j ^* ) is

Determined by.

Similarly, ^{the target pixel of (x *} ₀ (T-2), y ^* ₀ (T-2)) is set.
(X ^* ₀ (T-2), y ^* ₀ (T-2))
= (X ^* ₀ (T-1) + i ^* , y ^* ₀ (T-1) + j ^* ).

によって定まる。 In this case, (i ^* , j ^* ) is

Determined by.

In this way, in order to obtain the target pixel of the point sequence up to ^{(x *} ₀ (1), y ^* ₀ (1), 1) = (x ₀ , y _{0, 1), it is calculated T times in succession.} By doing
(X ^* ₀ (1), y ^* ₀ (1), 1) = (x ₀ , y ₀ , 1) can be obtained.

Next, from the arrival point (x ^* ₀ (T), y ^* ₀ (T), T) to (x ^* ₀ (1), y ^* ₀ (1), 1) = (x ₀ , y ₀ , 1) An example will be described of a method of calculating the distance from the camera 200 to an object by applying the point sequence up to the above to a moving image of a camera that captures the front of the vehicle in the traveling direction.

Five (M = 5) target pixels are extracted from the frame image at time t = 1. As T = 30, each target pixel is tracked by the method described above to obtain the arrival point of the frame image at time t = 30 (= T). FIG. 7 shows a locus (tracking locus) when the target pixel is tracked from time t = 1 to time t = 30 with one of the pixels in which the traffic light is reflected in the frame image at time t = 1 as the target pixel. It is an enlarged view shown in the frame image of time t = 30 (= T). FIG. 8 shows a locus (tracking locus) when five different traffic lights are set as objects, one of the pixels of the traffic light is set as each target pixel, and the five target pixels are tracked at the same time. Moreover, the frame image at the time t = 30 is shown.

For the five target pixels shown in FIG. 8, the fluctuation range of the tracking distance (the length of the locus distance from the coordinates of the start pixel to the coordinates of the end pixel of the tracking locus) on the frame image is measured. Of the five target pixel tracking distances, the number of pixels (minimum number of pixels) when the tracking distance is the minimum distance is 4.0 pixels, and the number of pixels (maximum number of pixels) when the tracking distance is the maximum distance is It was 30.52 pixels. The values of μ and γ described above are determined based on the number of pixels of these tracking distances. In the case of the present embodiment, 4.0 pixels indicating the minimum number of pixels correspond to the value of the minimum distance μ in the above-mentioned equation 2, and 30.52 pixels indicating the maximum number of pixels correspond to the value of the above-mentioned equation 3 in the above-mentioned equation 3. It corresponds to the maximum distance γ.
Minimum distance (μ) = 4.0 pixels, maximum distance (γ) = 30.52 pixels

Also, of the five objects (five objects corresponding to the five target pixels) shown in the frame image at time t = 30 (= T), by human inspection, in the real world. The distance from the camera 200 to the corresponding five objects (signal: object) is included in the range from the minimum distance of 50 m to the maximum distance of 100 m. These distances indicate the actual scale width from the object shown in the moving image to the camera 200. 50 m indicating the minimum distance _{corresponds to the value of Z N in} the above-mentioned equation 3, and 100 m indicating the maximum distance corresponds to the value _{of Z L in the above-mentioned equation 2.}
Minimum distance (Z _N ) = 50 m, maximum distance (Z _L ) = 100 m

In addition, based on the property of dynamic parallax described above, the farther the distance from the camera 200 to the object is (Z _L ), the shorter the tracking distance of the target pixel of the corresponding object (μ), and the distance from the camera 200 is increased. The closer the distance _{to the object (Z N} ), the longer the tracking distance of the target pixel of the corresponding object (γ). Therefore, μ _{corresponds to Z L} and γ corresponds to Z _N, and the relationship shown in Equations 2 and 3 is established.

By calculating the above-mentioned equations 4, 5, and 1 using these μ, γ, Z _L , and Z _N , the object reflected in the pixel points corresponding to the five target pixels to the camera 200 The actual distance of can be calculated. FIG. 8 shows the distance in the real world (distance from the camera 200 in the real world to each traffic light) calculated corresponding to the trajectories of the five target pixels. As shown in FIG. 8, variables a and b are calculated based on Equations 2 to 5 using _{the above-mentioned values of μ, γ, Z L} , and Z _{N, and the calculated a and b are converted into Equation 1.} By substituting, the value of the real-world distance from the object reflected in the coordinates of each target pixel to the camera 200 can be calculated.

Next, the selection / determination of the target pixel will be described. In principle, it is possible to set all the pixels of the frame image f (x, y, t) (however, 1 ≦ x ≦ X, 1 ≦ y ≦ Y) as target pixels. However, if the target of the target pixel is expanded too much, the amount of calculation for calculating the distance will increase. Therefore, it is preferable to select and reduce the number of target pixels so as to be appropriate. As an example of a method for appropriately selecting and reducing target pixels, a known region division method is used.

As one of the most promising methods among the existing area division methods, a method called a mean-shift method (intermediate value shift method) is known. The mean-shift method is a well-known area division method and is provided by a widely open library for computer vision called Open CV (Open Source Computer Vision Library). By applying the mean-shift method to the frame image, the image area is divided according to the object or the like reflected in the frame image based on the RGB value (color information) for each pixel of the frame image. Of the divided areas, the parts that are judged to be the same area are interpreted as having almost the same distance from the camera. In the present embodiment, the center of gravity point is obtained for each divided area (divided area), and each center of gravity point is determined as a target pixel. By determining the target pixel in this way, the distance can be calculated for each region where the distance from the camera 200 can be interpreted to be substantially the same, and the target pixel can be appropriately selected. Further, since the number of target pixels can be reduced to the number of divided areas, it is possible to reduce the calculation load.

FIG. 9 shows an image in which the mean-shift method is applied to a landscape image taken by a camera. By applying the mean-shift method, the landscape of the frame image is divided into multiple areas, and the same area is shown in the same color. The position of the center of gravity is obtained for each region divided in this way, and the pixel at the corresponding center of gravity is set as the target pixel.

FIG. 11A is a flowchart showing a method in which the CPU 104 calculates the distance from the camera 200 to the object based on the moving image in the front direction of the vehicle taken by the camera. The moving image for which the distance is calculated is the frame image f (x, y, t) (1 ≦ x ≦ X, 1 ≦ y ≦ Y, τ −T ≦) from the time t = τ−T to the time t = τ. t ≦ τ).

FIG. 10 is a block diagram showing each functional unit when the CPU 104 executes the distance calculation process based on the program read from the ROM 102. The CPU 104 has an area division unit (area division means) 301, a target pixel extraction unit (target pixel extraction means) 302, and a locus calculation unit (trajectory calculation means) 303 according to the processing contents shown in FIGS. 11A and 11B. , Functions as a moving pixel number measuring unit (moving pixel number measuring means) 304 and a distance calculating unit (distance calculating means) 305, and performs their respective processes.

First, the CPU 104 reads a frame image with a time t = τ−T from the recording unit 101, applies a mean-shift method to the read frame image, and performs area division processing of the frame image (FIG. 11 (a). ) S.1: Area division step, area division function). In this process, the CPU 104 functions as a region dividing unit (region dividing means) 301 that divides the frame image at time t = τ−T into a plurality of regions.

The CPU 104 obtains an area in which an object whose distance is measured from the camera 200 is projected, out of the area of the frame image divided into a plurality of areas. When there are M objects whose distances are to be measured, M areas are obtained. In addition, M is a number of 2 or more. Then, the CPU 104 obtains the coordinates of the center of gravity point for each of the M regions, and extracts the pixels of the coordinates of the obtained center of gravity points as the target pixels of the time t = τ−T for each of the M regions (S). .2: Target pixel extraction step, target pixel extraction function). In this process, the CPU 104 functions as a target pixel extraction unit (target pixel extraction means) 302 for extracting M target pixels.

The CPU 104 has a plurality of (M) target pixels determined in the frame image at time t = τ-T, and the CPU 104 has the target pixels from the frame image at time t = τ-T to the frame image at time t = τ. The locus is calculated by a tracking algorithm using a dynamic planning method for a two-dimensional image (S.3: locus calculation step, locus calculation function). ^{Then, the CPU 104 determines the coordinates (x *} ₀ (T), y ^* ₀ (T)) of the arrival point of the target pixel in the frame image at time t = τ. In this process, the CPU 104 sets the locus of the target pixel from the pixel position of the target pixel in the frame image at time t = τ −T to the pixel position of the target pixel in the frame image at time t = τ according to M target pixels. It functions as a locus calculation unit (trajectory calculation means) 303.

When the mean-shift method is applied to the frame image of time t = τ and the area division processing of the frame image of time t = τ is performed in the CPU 104, the frame image of time t = τ −T is used. The divided area to which the target pixel belongs and the divided area of the pixel point reached by the corresponding target pixel in the frame image at time t = τ usually coincide with each other.

The CPU 104 sets the length (number of pixels) of the tracking distance (movement distance) of the target pixel from the pixel position of the frame image at time t = τ −T to the pixel position of the frame image at time t = τ by M pieces. It measures for each target pixel (S.4: Moving pixel number measuring step, moving pixel number measuring function). In this process, the CPU 104 measures the number of moving pixels by measuring the moving distance of M target pixels as the number of moving pixels q _m (m = 1, 2, ..., M) in the frame image at time t = τ. It functions as a unit (moving pixel number measuring means) 304.

CPU104, of the moving pixel number q _m of the M frame image obtained, most mobile number of pixels of the smaller number of pixels and μ described above, and γ mentioned above the most number of moving a large number of pixels pixel. Furthermore, from the object moving pixel number is specified by the largest number of pixels of the target pixel to the camera 200, the distance of the real world in the time t = tau, and Z _n as determined by visual observation. Further, the distance in the real world from the object specified by the target pixel having the smallest number of moving pixels to the camera 200 at time t = τ is visually determined and defined as Z _L.

Then, the CPU 104 obtains the constant a and the constant b by substituting the values into the above-mentioned equations 4 and 5 using the set μ, γ, Z _N , and Z _{L, and obtains the constant a.} Using the constant b and the number of moving pixels q _m (m = 1, 2, ..., M) for each M target pixels, the time from the object corresponding to each target pixel to the camera 200. Calculate the distance in the real world at t = τ (S.5: distance calculation step, distance calculation function).

In this process, the CPU 104 performs the camera 200 from each object based on the constants a and b obtained based on _{the set μ, γ, Z N} , and Z _{L and the number of moving pixels q m for each target pixel.} It functions as a distance calculation unit (distance calculation means) 305 that calculates the distance in the real world up to.

In this way, the CPU 104 calculates the distance in the real world at the time t = τ from the object reflected in the moving image to the camera 200 based on the moving image taken by the camera 200. By identifying the divided areas (divided areas) in the frame image at time t = τ, what kind of object is reflected in each divided area, such as a traffic light, a signboard, a road, a wall, a tree, a building, etc. It is possible to classify whether or not it corresponds to an object, and the distance information from the camera 200 to each classified object can be obtained as a parameter. The parameters thus obtained can be used as important control parameters in, for example, automatic driving control of automobiles and the like.

The moving image distance calculation device and the moving image distance calculation program according to the present invention have been described in detail with reference to the drawings. However, the moving image distance calculation device and the moving image distance calculation program according to the present invention have been carried out. The present invention is not limited to the configuration example of the moving image distance calculation device 100 shown in the embodiment. In the target pixel tracking process described in the embodiment, a method of determining the target pixel based on the center of gravity point of the region divided by the mean-shift method has been described. However, the target pixel is not necessarily limited to the one determined based on the center of gravity point of the divided region.

For example, using the optimum cumulative value S (x, y, t) and the preset threshold value h,
S (x, y, t) / t ≦ h
Let R (x, y, t) be a region satisfying the above condition. A method of determining r (x ^* , y ^* , t) as the target pixel may be used, where the coordinates of the center of gravity of this region are r (x ^* , y ^{*, t).} Therefore, in the tracking process of the target pixel, r (x ^* , y ^* , t) is used to track t = 1, 2, ..., T. Here, the process of dividing the minimum cumulative value S (x, y, t) by the time t in order to obtain R (x, y, t) is the process of normalizing the minimum cumulative value by the length of time. Applicable. Further, the threshold value h is a constant determined empirically.

Even if the target pixel is determined by the center of gravity point of the divided area, there are pixels similar to the target pixel in the divided area. Therefore, a pixel having a similar integrated value also exists around the pixel position of the target pixel to be tracked. Therefore, the value of the normalized minimum cumulative value (S (x, y, t) / t) that is equal to or less than the threshold value h also indicates the divided region, and the coordinates of the center of gravity of this region are set as the target pixel. By making a decision, the reliability of the target pixel required by the tracking process will be increased.

Further, by tracking the coordinates of the center of gravity of this region with the region satisfying the condition of S (x, y, t) / t ≦ h as R (x, y, t), it is easy to visualize the back trace. Therefore, it is possible to easily determine whether the tracking is performed accurately.

Further, in the conventional tracking method, it is necessary to perform tracking processing individually for each object reflected in the moving image, but in the moving image distance calculation device and the moving image distance calculation program according to the present invention, it is shown in FIG. As described above, it is possible to simultaneously perform tracking processing on a plurality of objects (more specifically, a plurality of target pixels).

For example, the plurality of objects each of target pixels _{f (x m, y m,} t) ( where, m = 1,2, ···, M ) and. Here, m is a number for distinguishing the objects, and M is the total number of the objects.

とする。さらに、トラッキング処理の対象となるターゲットピクセルを、Ｓ（ｘ，ｙ，ｔ）／ｔの値が閾値ｈ以下となる複数の局所的領域の重心に決定する。 The recurrence formula of a dynamic programming for tracking target pixel of the object for each local distance _{d m (x, y, t} ) and _{d m (x, y, t} ) = | f (x m, _{y m, 1) -f (x} , y, t) |
And the minimum cumulative value S (x, y, t)

And. Further, the target pixel to be tracked is determined to be the center of gravity of a plurality of local regions where the value of S (x, y, t) / t is equal to or less than the threshold value h.

By determining the target pixel for each of a plurality of objects in this way, the distance from each object to the camera is determined by the path distance from the end to the start of the tracked target pixel locus (pixel distance of the locus). Can be used for each object.

Further, in the moving image distance calculation device 100 according to the embodiment, as shown in FIG. 11A, the CPU 104 changes with time from the past to the present (time t = τ−T to time t =). A case where tracking processing of a target pixel is performed based on a frame image of a moving image (which changes up to τ) has been described. However, when the tracking process of the target pixel is performed, it is also possible to perform the tracking process of the target pixel in the frame image of the moving image by tracing back the time from the time t = τ to the past time t = τ−T. Is.

FIG. 11B is a flowchart showing the processing contents when the CPU 104 performs the tracking process of the target pixel by tracing back the time t from the time t = τ to the time t = τ−T. The CPU 104 divides a frame image at time t = τ by applying a mean-shift method (S.11: region division step, region division function).

Then, the CPU 104 obtains M areas in which the object is reflected from the obtained divided areas, and extracts the pixel of the center of gravity point as the target pixel for each of the M areas (S.12: Target pixel extraction step). , Target pixel extraction function). For example, let one pixel of the extracted M target pixels be (x ^* ₀ (τ), y ^* ₀ (τ)).

The CPU 104 performs tracking processing of the target pixel based on the moving image from the time t = τ to the time t = τ−T (S.13: trajectory calculation step, trajectory calculation function). By this tracking process, the locus of the target pixel is obtained by the time series going back to the past from the time t = τ to the time t = τ−T. Let the arrival point of one of the M target pixels in the frame image at time t = τ-T be (x ^* ₀ (τ-T), y ^* ₀ (τ-T)). ^{The CPU 104 sets the cumulative dynamic parallax value q (x *} ₀ (τ), y ^* ₀ (τ)) of the target pixel according to the locus from the time t = τ to the time t = τ−T.
q (x ^* ₀ (τ), y ^* ₀ (τ))
= | (X ^* ₀ (τ) -y ^* ₀ (τ))-(x ^* ₀ (τ-T), y ^* ₀ (τ-T)) |
Ask more. The value q (x ^* ₀ (τ), y ^* ₀ (τ)) of this cumulative dynamic parallax corresponds to the _{number of moving pixels q m already described.} Therefore, the CPU 104 measures the number of moving pixels by this process (S.14: moving pixel number measuring step, moving pixel number measuring function).

Then, the CPU 104 calculates the distance of the object reflected in the frame image at time t = τ based on the obtained cumulative dynamic parallax value q (x ^* ₀ (τ), y ^* _{0 (τ)).} (S.15: Distance calculation step, distance calculation function). The real-world distance from the object to the camera 200 at time t = τ is
d (x ^* ₀ (τ), y ^* ₀ (τ))
= A · exp (−bq (x ^* ₀ (τ), y ^* ₀ (τ)))
Can be obtained by.

This equation corresponds to Z = a · exp (−bq) shown in Equation 1.

In this way, the time is traced back from the time t = τ to the past time t = τ−T, and the tracking process of the target pixel in the frame image of the moving image is performed to obtain the distance in the real world. There are two advantages.

First, the divided region in the frame image at time t = τ tends to be a relatively wide region because the distance to the object reflected in the frame image is short. This has the advantage that the target pixel can be determined based on a more stable split area.

Next, since the target pixel is tracked by tracing back the time from the time t = τ to the past time t = τ−T, the phenomenon that the target pixel is out of the frame of the moving image even if the time is traced back. Has the advantage of being less likely to occur. In the moving image that changes from time t = τ-T to time t = τ, the target pixel existing in the frame image of time t = τ-T changes to time t = τ as the camera 200 progresses. It may be out of the frame image before it becomes, and it may not exist in the frame image at time t = τ. However, when tracking the target pixel by tracing back the time from the time t = τ to the past time t = τ−T, the target pixel tends to move toward the center of the screen. Even if the amount of movement of the target pixel is small, it is highly likely that it exists in the frame image.

Similarly, it is highly possible that the divided region in which the target pixel is determined also exists in the frame image at time t = τ−T. Therefore, when tracking the target pixel by tracing back the time from the time t = τ to the past time t = τ−T, the target pixel and the divided area deviate from the frame image. The advantage is that the need to deal with it is reduced.

When tracking the target pixel by going back in time, it cannot be guaranteed that the terminal pixel will always be the same pixel as the correct target pixel, but in this regard, time t = τ−T to time t = τ. The same applies even when tracking the target pixel over time.

Further, in the moving image distance calculation device 100 according to the embodiment, the following method can be used in order to stably perform the tracking process of the target pixel.

First, in the image distance calculation device 100 according to the embodiment, a case where a center of gravity point is obtained for each divided area (divided area) and each center of gravity point is determined as a target pixel has been described. However, when the shape of the divided region is, for example, an L-shape, the center of gravity may be outside the divided region. In such a case, the coordinates outside the divided area may be determined as the target pixel. Therefore, numbers are added to the pixels in the area in order from the end, and the number corresponds to half the value of all the pixels in the area (corresponds to the middle number among the numbers of the pixels in the area). Pixel is determined as the target pixel. By determining the target pixel in this way, the pixel in the divided region can be reliably determined as the target pixel.

Next, the coordinates of the determined target pixel are set to S. in FIG. 11 (b). As set in 12, when (x ^* ₀ (τ), y ^* ₀ (τ)), the value of the target pixel is not determined by the value of only one pixel, but above and below the coordinates of the target pixel. It is also possible to determine by the average value of the values of the left and right pixels.

・・・式８
により値を決定することができる。このように複数の座標の値の平均値によりターゲットピクセルの値を決定することにより、より安定した値を得ることが可能になる。 ^{For example, the value P (x *} ₀ (x * 0 (x * 0)) of the target pixel is used by using the average value of five coordinate points including the coordinates of the target pixel (x ^* ₀ (τ), y ^* _{0 (τ)).} When determining τ), y ^* ₀ (τ)),

... Equation 8
The value can be determined by. By determining the value of the target pixel based on the average value of the values of the plurality of coordinates in this way, a more stable value can be obtained.

Further, in the above-mentioned calculation of the local distance, in the image distance calculation device 100 according to the embodiment, the value of the target pixel is f (x ₀ , y ₀ , 1), and the local distance is the value of the target pixel and the time. Calculated by the eucritic distance from the value of the target pixel of t,
Local distance d (x, y, t) = | f (x ₀ , y ₀ , 1) -f (x, y, t) |
I explained the case of requesting by.

・・・式９
により求めることができる。 However, the calculation of the local distance is not limited to this method. For example, it is also possible to obtain the local distance by calculating the distance value using the cosine correlation function of two pixels. That is, the local distance d (x, y, t) is set to

・ ・ ・ Equation 9
Can be obtained by.

<F (x, y, t), P (x ^* ₀ (τ), y ^* ₀ (τ))> shown in the numerator of Equation 9 indicates the inner product of RGB vectors. Since the correlation value always takes a value between zero and 1, the value obtained by Equation 9 is stably obtained with respect to the absolute fluctuation of the pixel value. Therefore, the local distance calculated by the equation 9 has a feature that it is not easily affected by noise or the like that may occur in the frame image.

The parameter of the time t in the case of the equation 9 is τ−T ≦ t ≦ τ as already explained with reference to FIG. 11B, but the progress of the calculation is from the time τ to the time τ−T. It is performed by advancing the time t so as to go back to the past. That is, the time t progresses as time t = τ, τ-1, τ-2, ..., τ−T, and the time t to reach the target pixel is time t = τ−T.

Further, by visualizing the back trace in the tracking process of the target pixel, it becomes easy to judge whether the tracking is performed accurately. In this respect, as described above, the region satisfying the condition of S (x, y, t) / t ≦ h is set as R (x, y, t), and the coordinates of the center of gravity of this region are tracked. , It is possible to facilitate the visualization of the back trace.

100 ... Moving image distance calculation device 101 ... Recording unit 102 ... ROM
103 ... RAM
104 ... CPU (control means, area division means, target pixel extraction means, distance calculation means, locus calculation means, moving pixel number measuring means, distance calculation means)
200 ... Camera 210 ... Monitor 301 ... Area division unit (area division means)
302… Target pixel extraction unit (target pixel extraction means)
303 ... Trajectory calculation unit (trajectory calculation means)
304 ... Moving pixel number measuring unit (moving pixel number measuring means)
305… Distance calculation unit (distance calculation means)

Claims (8)

The front view in the moving direction is reflected in the moving image based on the moving image taken from the time t = τ-T (however, τ> 0, T> 0) to the time t = τ by one moving camera. A moving image distance calculation device that calculates the distance from an object to the camera.
One of the pixels of the object reflected in the frame image when the time t = τ−T of the moving image is set as the target pixel, and the target pixel is set for each of M (M ≧ 2) different objects from the frame image. Target pixel extraction means to extract and
Based on the plurality of frame images from time t = τ−T to time t = τ, the loci of the coordinates of the M target pixels extracted by the target pixel extraction means are targeted for a two-dimensional image. A locus calculation means for calculating in chronological order from the frame image at time t = τ −T to the frame image at time t = τ based on the dynamic planning method.
_{Based on the M loci calculated by the locus calculation means, the number of moving pixels q m} (m =) from the coordinates of the frame image at time t = τ −T to the coordinates of the frame image at time t = τ. A moving pixel number measuring means for measuring 1, 2, ..., M) for each of the M target pixels,
Wherein among the mobile M-number of the number of pixels moved which is measured by the number measuring unit pixels q _m, the moving pixel number is the smallest number of pixels as mu, the number of pixels the number of moving pixels is highest and gamma, the M the nearest distance among the distances from the object to be identified respectively by the target pixel to the camera and Z _n, among the objects identified respectively by the M of the target pixel of the distance to the camera _{Let Z L be} the farthest distance, and let constant a and constant b be
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (γ-μ)) log (Z _L / Z _N )
Calculated by
_{Let Z m} (m = 1, 2, ..., M) be the distance from the object corresponding to each of the M target pixels to the camera at time t = τ _{, and set the distance Z m} as described above. Based on the constant a and the constant b and the number of M moving pixels q _m ,
Z _m = a · exp (-bq _m )
A moving image distance calculation device characterized by having a distance calculation means calculated by The front view in the moving direction is reflected in the moving image based on the moving image taken from the time t = τ-T (however, τ> 0, T> 0) to the time t = τ by one moving camera. A moving image distance calculation device that calculates the distance from an object to the camera.
One of the pixels of the object reflected in the frame image when the time t = τ of the moving image is set as the target pixel, and the target pixel is extracted for each of M (M ≧ 2) different objects from the frame image. Target pixel extraction method and
Based on the plurality of frame images from time t = τ−T to time t = τ, the loci of the coordinates of the M target pixels extracted by the target pixel extraction means are targeted for a two-dimensional image. Based on the dynamic planning method, a locus calculation means for calculating in chronological order retroactively from the frame image at time t = τ to the frame image at time t = τ −T.
_{Based on the M loci calculated by the locus calculation means, the number of moving pixels q m} (m =) from the coordinates of the frame image at time t = τ −T to the coordinates of the frame image at time t = τ. A moving pixel number measuring means for measuring 1, 2, ..., M) for each of the M target pixels,
Wherein among the mobile M-number of the number of pixels moved which is measured by the number measuring unit pixels q _m, the moving pixel number is the smallest number of pixels as mu, the number of pixels the number of moving pixels is highest and gamma, the M the nearest distance among the distances from the object to be identified respectively by the target pixel to the camera and Z _n, among the objects identified respectively by the M of the target pixel of the distance to the camera _{Let Z L be} the farthest distance, and let constant a and constant b be
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (γ-μ)) log (Z _L / Z _N )
Calculated by
_{Let Z m} (m = 1, 2, ..., M) be the distance from the object corresponding to each of the M target pixels to the camera at time t = τ _{, and set the distance Z m} as described above. Based on the constant a and the constant b and the number of M moving pixels q _m ,
Z _m = a · exp (-bq _m )
A moving image distance calculation device characterized by having a distance calculation means calculated by The target pixel extraction means has a region dividing means for dividing the frame image into a plurality of regions by applying a mean-shift method to the frame image used for extracting the target pixels.
The target pixel extraction means has M (M ≧ 2) objects of different objects reflected in the area divided by the area dividing means, with the pixel of the center of gravity of the area in which the object is reflected as the target pixel. The moving image distance calculation device according to claim 1 or 2, wherein the target pixel is extracted for each region. The target pixel extraction means has a region dividing means for dividing the frame image into a plurality of regions by applying a mean-shift method to the frame image used for extracting the target pixels.
The target pixel extraction means adds numbers in order from the end to the pixels in the area divided by the area dividing means, and the pixel corresponding to the middle number among the numbers of the pixels in the area is the target pixel. The moving image distance calculation device according to claim 1 or 2, wherein the target pixels are extracted for each of M (M ≧ 2) regions in which different objects are reflected. The front view in the moving direction is reflected in the moving image based on the moving image taken from the time t = τ-T (however, τ> 0, T> 0) to the time t = τ by one moving camera. A moving image distance calculation program for calculating the distance from an object to the camera.
As a control means,
One of the pixels of the object reflected in the frame image when the time t = τ−T of the moving image is set as the target pixel, and the target pixel is set for each of M (M ≧ 2) different objects from the frame image. Target pixel extraction function to extract and
Based on the plurality of frame images from time t = τ−T to time t = τ, the loci of the coordinates of the M target pixels extracted by the target pixel extraction function are targeted for a two-dimensional image. Based on the dynamic planning method, a trajectory calculation function for calculating in chronological order from the frame image at time t = τ-T to the frame image at time t = τ, and
_{Based on the M loci calculated by the locus calculation function, the number of moving pixels q m} (m =) from the coordinates of the frame image at time t = τ −T to the coordinates of the frame image at time t = τ. A moving pixel number measurement function that measures 1, 2, ..., M) for each of the M target pixels,
Wherein among the moving pixel number measurement function by the measured M number of the mobile number of pixels q _m, the moving pixel number is the smallest number of pixels as mu, the number of pixels the number of moving pixels is highest and gamma, the M the nearest distance among the distances from the object to be identified respectively by the target pixel to the camera and Z _n, among the objects identified respectively by the M of the target pixel of the distance to the camera _{Let Z L be} the farthest distance, and let constant a and constant b be
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (γ-μ)) log (Z _L / Z _N )
Calculated by
_{Let Z m} (m = 1, 2, ..., M) be the distance from the object corresponding to each of the M target pixels to the camera at time t = τ _{, and set the distance Z m} as described above. Based on the constant a and the constant b and the number of M moving pixels q _m ,
Z _m = a · exp (-bq _m )
A moving image distance calculation program characterized by realizing a distance calculation function calculated by The front view in the moving direction is reflected in the moving image based on the moving image taken from the time t = τ-T (however, τ> 0, T> 0) to the time t = τ by one moving camera. A moving image distance calculation program for calculating the distance from an object to the camera.
As a control means,
One of the pixels of the object displayed in the frame image when the time t = τ of the moving image is set as the target pixel, and the target pixel is extracted from the frame image for each of M (M ≧ 2) different objects. Target pixel extraction function and
Based on the plurality of frame images from time t = τ−T to time t = τ, the loci of the coordinates of the M target pixels extracted by the target pixel extraction function are targeted for a two-dimensional image. Based on the dynamic planning method, a trajectory calculation function that calculates from the frame image at time t = τ to the frame image at time t = τ −T in chronological order retroactively.
_{Based on the M loci calculated by the locus calculation function, the number of moving pixels q m} (m =) from the coordinates of the frame image at time t = τ −T to the coordinates of the frame image at time t = τ. A moving pixel number measurement function that measures 1, 2, ..., M) for each of the M target pixels,
Wherein among the moving pixel number measurement function by the measured M number of the mobile number of pixels q _m, the moving pixel number is the smallest number of pixels as mu, the number of pixels the number of moving pixels is highest and gamma, the M the nearest distance among the distances from the object to be identified respectively by the target pixel to the camera and Z _n, among the objects identified respectively by the M of the target pixel of the distance to the camera _{Let Z L be} the farthest distance, and let constant a and constant b be
a = Z _L · exp ((μ / (γ-μ)) log (Z _L / Z _N ))
b = (1 / (γ-μ)) log (Z _L / Z _N )
Calculated by
_{Let Z m} (m = 1, 2, ..., M) be the distance from the object corresponding to each of the M target pixels to the camera at time t = τ _{, and set the distance Z m} as described above. Based on the constant a and the constant b and the number of M moving pixels q _m ,
Z _m = a · exp (-bq _m )
A moving image distance calculation program characterized by realizing a distance calculation function calculated by To the control means
By applying the mean-shift method to the frame image used for extracting the target pixel in the target pixel extraction function, a region division function for dividing the frame image into a plurality of regions is realized.
In the target pixel extraction function, among the areas divided by the area division function, M (M ≧ 2) in which different objects are reflected, with the pixel at the center of gravity of the area in which the object is reflected as the target pixel. The moving image distance calculation program according to claim 5 or 6, wherein the target pixel is extracted for each region. To the control means
By applying the mean-shift method to the frame image used for extracting the target pixel in the target pixel extraction function, a region division function for dividing the frame image into a plurality of regions is realized.
In the target pixel extraction function, numbers are added in order from the end to the pixels in the region divided by the region division function, and the pixel corresponding to the middle number among the pixel numbers in the region is the target pixel. The moving image distance calculation program according to claim 5 or 6, wherein the target pixels are extracted for each of M (M ≧ 2) regions in which different objects are reflected.

Images