WO2019239460A1 - Camera calibration device - Google Patents

Abstract

A camera calibration device (100) comprises: a reference point position information acquisition unit (21) that acquires reference point position information indicating the position of a reference point for an imaging image (I) from a camera (1); a reference point coordinate information acquisition unit (22) that acquires reference point coordinate information indicating the coordinate values corresponding to the position of the reference point in actual space; a horizontal line position information acquisition unit (23) that acquires horizontal line position information indicating the position of a horizontal line or a horizon line in the imaging image (I); and an external parameter calculation unit (24) that calculates the orientation angle (ω, φ, κ) of the camera (1) on the basis of the position of the reference point in the imaging image (I), the coordinate values corresponding to the position of the reference point in the actual space, and the position of the horizontal line or horizon line in the imaging image (I).

Description

Camera calibration device

The present invention relates to a camera calibration device.

Conventionally, when an object such as a moving object (for example, a ship) is captured in an image captured by a camera such as a so-called “live camera”, the position of the object in real space is determined according to the position of the object in the captured image. A method of identifying has been developed. This identification method uses parameters corresponding to the angle of view and lens distortion of the camera (so-called “internal parameters”) and parameters corresponding to the installation position and orientation angle of the camera (so-called “external parameters”) in advance. Calibration is required.

On the other hand, a technique for calculating the attitude angle of the camera based on the positions of a plurality of GCPs (Ground Control Points) in the real space and the positions of the plurality of GCPs in the image captured by the camera has been developed ( For example, refer nonpatent literature 1.). Each of the plurality of GCPs corresponds to, for example, a structure on the ground or water whose coordinate values (hereinafter referred to as “absolute coordinate values”) indicating positions in real space are known and constant.

R. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," in IEEE Journal on Robotics and Automation, vol. 3, no. 323-, pp. 344, August 1987.

The conventional technique described in Non-Patent Document 1 and the like uses a plurality of GCPs for calculating the attitude angle. However, depending on the installation location of the camera, there may be only one or zero GCP within the imaging range of the camera. That is, a plurality of GCPs may not be prepared in advance. The conventional technique described in Non-Patent Document 1 and the like has a problem that the posture angle cannot be calculated in such a case.

The present invention has been made to solve the above-described problems, and an object of the present invention is to reduce the number of GCPs used for calculating the attitude angle by the camera calibration device.

The camera calibration device of the present invention includes a reference point position information acquisition unit that acquires reference point position information indicating the position of a reference point in an image captured by the camera, and a reference point that indicates a coordinate value corresponding to the position of the reference point in real space. Reference point coordinate information acquisition unit for acquiring coordinate information, horizontal line position information acquisition unit for acquiring horizontal line position information indicating the position of the horizontal line or horizon in the captured image, the position of the reference point in the captured image, and the reference point in real space An external parameter calculation unit that calculates a camera posture angle based on the coordinate value corresponding to the position and the position of the horizontal line or the horizon in the captured image.

Alternatively, the camera calibration device of the present invention has a horizontal line position information acquisition unit that acquires horizontal line position information indicating the position of a horizontal line or a horizon in a captured image by the camera, and a moving object is captured at a predetermined position in the captured image. A moving body area information acquisition unit that acquires moving body area information indicating an area corresponding to the moving body in the captured image, and a position of the moving body in real space when the moving body is captured at a predetermined position A moving body coordinate information acquisition unit that acquires moving body coordinate information including coordinate values, and a coordinate value corresponding to the position of the horizontal line or the horizon in the captured image, the position of the moving body in the captured image, and the position of the moving body in real space. And an external parameter calculation unit for calculating a camera attitude angle.

According to the present invention, since it is configured as described above, it is possible to reduce the number of GCPs used for calculating the attitude angle by the camera calibration device.

FIG. 3 is a block diagram illustrating a main part of a computer including the camera calibration device according to the first embodiment. FIG. 2A is a block diagram illustrating a hardware configuration of a computer including the camera calibration device according to the first embodiment. FIG. 2B is a block diagram illustrating another hardware configuration of the computer including the camera calibration device according to Embodiment 1. 3 is a flowchart showing an operation of a computer including the camera calibration device according to the first embodiment. It is explanatory drawing which shows the example of UI when reference point position information, reference point coordinate information, and horizontal line position information are input. It is explanatory drawing which shows the example of UI when reference point position information, reference point coordinate information, and horizontal line position information are input. It is explanatory drawing which shows the example of a world coordinate system. It is explanatory drawing which shows the example of a camera coordinate system. It is a block diagram which shows the principal part of the computer containing the camera calibration apparatus which concerns on Embodiment 2. FIG. 6 is a flowchart illustrating an operation of a computer including a camera calibration device according to a second embodiment. 6 is a flowchart illustrating an operation of a computer including a camera calibration device according to a second embodiment.

Hereinafter, in order to explain the present invention in more detail, modes for carrying out the present invention will be described with reference to the accompanying drawings.

Embodiment 1 FIG.
FIG. 1 is a block diagram illustrating a main part of a computer including a camera calibration device according to the first embodiment. A camera calibration apparatus 100 according to the first embodiment will be described with reference to FIG.

The camera 1 is, for example, a live camera, a surveillance camera, or a security camera. That is, the captured image I by the camera 1 is a moving image. The camera 1 includes an image sensor such as a visible light camera, an infrared camera, or a distance image sensor (a so-called “TOF (Time of Flight) camera”).

The imaging control unit 11 controls the operation of the camera 1 and acquires image data indicating the captured image I from the camera 1. The imaging control unit 11 outputs the acquired image data, that is, image data indicating the captured image I to the input / output control unit 12.

The input / output control unit 12 receives input of information (hereinafter referred to as “reference point position information”) indicating the position of the GCP (hereinafter referred to as “reference point”) in the captured image I. The input / output control unit 12 receives an input of coordinate values corresponding to the position of the reference point in the real space, that is, information indicating the absolute coordinate value of the reference point (hereinafter referred to as “reference point coordinate information”). . The absolute coordinate value of the reference point is, for example, a latitude value, a longitude value, and an altitude value in a so-called “world coordinate system”. The input / output control unit 12 receives input of information indicating the position of the horizontal line in the captured image I (hereinafter referred to as “horizontal line position information”). Hereinafter, the reference point position information and the horizontal line position information are collectively referred to as “position information”.

The input / output control unit 12 causes the display device 2 to display a screen for inputting reference point position information, reference point coordinate information, and horizontal line position information (hereinafter referred to as “input screen”). The input screen includes a captured image I by so-called “live playback”. The reference point position information, the reference point coordinate information, and the horizontal line position information are input by operating the operation input device 3 on the input screen while the input screen is displayed on the display device 2. The display device 2 is configured by, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The operation input device 3 is composed of, for example, a keyboard and a mouse.

The reference point position information acquisition unit 21 acquires the input reference point position information when the input / output control unit 12 receives the input of the reference point position information. The reference point coordinate information acquisition unit 22 acquires the input reference point coordinate information when the input / output control unit 12 receives an input of the reference point coordinate information. The horizontal line position information acquisition unit 23 acquires the input horizontal line position information when the input / output control unit 12 receives input of horizontal line position information.

The external parameter calculation unit 24 is acquired by the reference point position information acquired by the reference point position information acquisition unit 21, the reference point coordinate information acquired by the reference point coordinate information acquisition unit 22, and the horizontal line position information acquisition unit 23. The posture angles ω, φ, κ of the camera 1 are calculated using the horizontal line position information. That is, the external parameter calculation unit 24 calculates the attitude angles ω, φ, and κ of the camera 1 based on the position of the reference point in the captured image I, the absolute coordinate value of the reference point, and the position of the horizontal line in the captured image I. Is. Here, ω is a posture angle of the camera 1 with respect to the roll direction, φ is a posture angle of the camera 1 with respect to the pitch direction, and κ is a posture angle of the camera 1 with respect to the yaw direction. A specific example of the method of calculating the posture angles ω, φ, κ by the external parameter calculation unit 24 will be described later.

The input / output control unit 12 executes control for outputting the posture angles ω, φ, and κ calculated by the external parameter calculation unit 24. More specifically, the input / output control unit 12 causes the display device 2 to display an image including the calculated posture angles ω, φ, and κ.

The reference point position information acquisition unit 21, the reference point coordinate information acquisition unit 22, the horizon position information acquisition unit 23, and the external parameter calculation unit 24 constitute a main part of the camera calibration apparatus 100. The imaging control unit 11, the input / output control unit 12, and the camera calibration device 100 constitute a main part of the computer 200.

Next, the hardware configuration of the main part of the computer 200 will be described with reference to FIG.

As shown in FIG. 2A, the computer 200 has a processor 31 and a memory 32. The memory 32 causes the computer 200 to function as the imaging control unit 11, the input / output control unit 12, the reference point position information acquisition unit 21, the reference point coordinate information acquisition unit 22, the horizontal line position information acquisition unit 23, and the external parameter calculation unit 24. A program for storing is stored. When the processor 31 reads out and executes the program stored in the memory 32, the imaging control unit 11, the input / output control unit 12, the reference point position information acquisition unit 21, the reference point coordinate information acquisition unit 22, and the horizontal line position information acquisition. The functions of the unit 23 and the external parameter calculation unit 24 are realized.

Alternatively, the computer 200 may include a processing circuit 33 as shown in FIG. 2B. In this case, the processing circuit 33 realizes the functions of the imaging control unit 11, the input / output control unit 12, the reference point position information acquisition unit 21, the reference point coordinate information acquisition unit 22, the horizontal line position information acquisition unit 23, and the external parameter calculation unit 24. It may be.

Alternatively, the computer 200 may include a processor 31, a memory 32, and a processing circuit 33 (not shown). In this case, some of the functions of the imaging control unit 11, the input / output control unit 12, the reference point position information acquisition unit 21, the reference point coordinate information acquisition unit 22, the horizontal line position information acquisition unit 23, and the external parameter calculation unit 24 The functions may be realized by the processor 31 and the memory 32, and the remaining functions may be realized by the processing circuit 33.

The processor 31 uses, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor).

The memory 32 uses, for example, at least one of a semiconductor memory, a magnetic disk, an optical disk, or a magneto-optical disk. More specifically, the memory 32 includes a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory EMM, an EEPROM (Electrically EMM). State Drive), HDD (Hard Disk Drive), FD (Floppy Disk), CD (Compact Disc), DVD (Digital Versatile Disc), MO (Magneto-Optical) or MD (MiniDis) Taiko It is.

The processing circuit 33 may be, for example, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-LargeSemi-ChemicalSigleSigureSigureSigureSigureSigureSigureSigureSigureSigureSigureSigure). Using at least one of them.

Next, the operation of the computer 200 will be described with reference to the flowchart of FIG. Note that the imaging control unit 11 is executing a process of acquiring image data indicating the captured image I from the camera 1 and outputting the acquired image data to the input / output control unit 12.

First, in step ST <b> 1, the input / output control unit 12 starts display of an input screen by the display device 2. The input screen includes a captured image I obtained by live reproduction.

Next, in step ST2, the input / output control unit 12 accepts input of one or more reference point position information according to the operation of the operation input device 3 on the input screen. Next, in step ST3, the reference point position information acquisition unit 21 acquires the input reference point position information.

Next, in step ST4, the input / output control unit 12 accepts input of one or more reference point coordinate information according to the operation of the operation input device 3 on the input screen. Next, in step ST5, the reference point coordinate information acquisition unit 22 acquires the input reference point coordinate information.

Next, in step ST6, the input / output control unit 12 accepts input of one or more pieces of horizontal line position information in accordance with the operation of the operation input device 3 on the input screen. Next, in step ST7, the horizontal line position information acquisition unit 23 acquires the input horizontal line position information.

Next, in step ST8, the input / output control unit 12 determines whether or not three or more pieces of position information have been input. When the number of pieces of input position information is two or less (step ST8 “NO”), the processing of the computer 200 returns to step ST6. On the other hand, when the number of input position information is three or more (step ST8 “YES”), the processing of the computer 200 proceeds to step ST9.

Next, in step ST9, the external parameter calculation unit 24 uses the reference point position information acquired by the reference point position information acquisition unit 21, the reference point coordinate information acquired by the reference point coordinate information acquisition unit 22, and the horizontal line position information. The posture angles ω, φ, κ of the camera 1 are calculated using the horizontal line position information acquired by the acquisition unit 23. A specific example of the method of calculating the posture angles ω, φ, κ by the external parameter calculation unit 24 will be described later.

Next, in step ST10, the input / output control unit 12 executes control to output the posture angles ω, φ, κ calculated by the external parameter calculation unit 24. More specifically, the input / output control unit 12 causes the display device 2 to display an image including the calculated posture angles ω, φ, and κ.

Next, a specific example of a UI (User Interface) when reference point position information, reference point coordinate information, and horizontal line position information are input will be described with reference to FIG. The number of input reference point position information is one, the number of input reference point coordinate information is one, and the number of input horizontal line position information is two. And

First, the user clicks the position of the reference point in the captured image I in the input screen using the mouse of the operation input device 3 (see (A) in FIG. 4A). Then, a circular marker image M1 corresponding to the clicked position is superimposed and displayed on the captured image I, and an input form for the absolute coordinate value of the reference point is popped up (see (B) of FIG. 4A). . The user inputs the absolute coordinate value of the reference point using the keyboard of the operation input device 3.

Next, the user uses the mouse of the operation input device 3 to press the “Next” button displayed in the lower right part of the input screen. Thereby, the input of the reference point position information and the reference point coordinate information is completed.

Next, the user clicks the position of the horizontal line in the captured image I in the input screen using the mouse of the operation input device 3 (see (C) in FIG. 4A). Then, a circular marker image M2 corresponding to the clicked position is superimposed on the captured image I (see (D) in FIG. 4B).

Next, the user clicks another position of the horizontal line in the captured image I in the input screen using the mouse of the operation input device 3 (see (E) in FIG. 4B). Then, a circular marker image M3 corresponding to the clicked position is displayed superimposed on the captured image I (see (F) in FIG. 4B).

Next, the user uses the mouse of the operation input device 3 to press the “complete” button displayed in the lower right part of the input screen. Thereby, the input of the horizontal line position information is completed.

Note that the color of the marker image M1 corresponding to the position of the reference point may be different from the colors of the marker images M2 and M3 corresponding to the position of the horizontal line. For example, the marker image M1 may be red and the marker images M2 and M3 may be yellow.

Next, a specific example of a method for calculating the posture angles ω, φ, κ by the external parameter calculation unit 24 will be described.

Usually, unless the altitude at the installation position of the camera 1 is extremely high, the imaging range by the camera 1 is about several kilometers to several tens of kilometers at maximum. For this reason, the distance from the installation position of the camera 1 to the horizontal line imaged in the captured image I is a sufficiently small value with respect to the radius of the earth. For this reason, the following calculation is executed assuming that the shape of the earth is a perfect circle.

First, the external parameter calculation unit 24 calculates the coordinate value corresponding to the installation position of the camera 1 in real space, that is, the absolute coordinate value (b, l, h) of the camera 1. Here, b is a latitude value in the world coordinate system, l is a longitude value in the world coordinate system, and h is an elevation value in the world coordinate system. FIG. 5 shows an example of the world coordinate system, more specifically, an example of the earth center coordinate system. Since the calculation method of the absolute coordinate value (b, l, h) is well known, detailed description is omitted. Instead of calculating the absolute coordinate value (b, l, h) by the external parameter calculation unit 24, information indicating the absolute coordinate value (b, l, h) is stored in the external parameter calculation unit 24 in advance. It may be.

Next, the external parameter calculation unit 24 calculates the center of the earth from the point corresponding to the absolute coordinate value (b, l, 0) having the same latitude value and longitude value as the absolute coordinate value (b, l, h) of the camera 1. The distance r1 to the part is calculated. The external parameter calculation unit 24 calculates a geoid height r2 corresponding to (b, l) of the absolute coordinate values (b, l, h) of the camera 1. The external parameter calculation unit 24 calculates a virtual earth radius r by r = r1 + r2. Since the calculation method of distance r1 and geoid height r2 is well-known, detailed description is abbreviate | omitted.

Here, the following equation (1) is established in the so-called “camera coordinate system”. FIG. 6 shows an example of the camera coordinate system. At this time, the line-of-sight vector VE in the camera coordinate system is expressed by the following equation (4) by the following equations (2) and (3). The line-of-sight vector VE in the camera coordinate system is a vector from the center of the camera 1 to a position corresponding to each pixel of the captured image I.

As shown in the following equation (5), the line-of-sight vector VE in the camera coordinate system is multiplied by a 3 × 3 conversion matrix to convert the line-of-sight vector VE in the camera coordinate system into a line-of-sight vector VE ′ in the world coordinate system. Is done. If the posture angles ω, φ, κ are set to known predetermined values, this transformation matrix can be calculated, and the following equation (6) is obtained.

When the line-of-sight vector VE ′ is in contact with the earth, the distance between the vector having the direction vector v and the center (0, 0, 0) of the earth passing through the installation position of the camera 1 is the radius r of the earth. Is equal to For this reason, the following formula | equation (7) is materialized. Further, the following equation (8) is obtained by the equational deformation of the equation (7).

In Expression (8), variables other than the vector v are constants, and the vector v is expressed by a linear expression of z ′. Therefore, this equation becomes a quadratic expression of z ′, and the solution is uniquely obtained using the solution formula of the quadratic equation.

Here, z ′ has an origin (that is, 0) corresponding to the center of the camera 1, a positive direction corresponding to the upper direction of the captured image I, and a lower direction of the captured image I. It is a coordinate value in a coordinate system having a corresponding negative direction. Therefore, the coordinate value v in the captured image I is calculated by the following equation (9). Thereby, the coordinate value v corresponding to the position of the horizontal line in the captured image I can be calculated.

V = -z '+ (height-1) / 2 (9)

As described above, the external parameter calculation unit 24 has already calculated the absolute coordinate values (b, l, h), or information indicating the absolute coordinate values (b, l, h) is stored in the external parameter calculation unit 24. ing. For this reason, the installation position of the camera 1 is known. Therefore, based on the above principle, it is possible to calculate the coordinate value v when the posture angles ω, φ, κ are set to known predetermined values, that is, the position of the horizontal line in the captured image I in this case. .

Therefore, the external parameter calculation unit 24 reduces the posture angle ω, φ, and the like so that the difference value between the horizontal line position obtained by the calculation and the actual horizontal line position (that is, the horizontal line position indicated by the horizontal line position information) becomes small. The iterative calculation is performed while changing the value of the parameter corresponding to κ. As a result, the posture angles ω, φ, κ corresponding to the parameter values when the difference value becomes the smallest, that is, the actual posture angles ω, φ, κ can be approximately obtained.

More specifically, the external parameter calculation unit 24 performs iterative calculation so as to minimize the objective function shown in the following equation (10) by a gradient method such as the least square method or the Levenberg Marquardt method.

Xi in Expression (10) is a coordinate value corresponding to the position of the i-th reference point in the captured image I, and indicates an actual measurement value of the coordinate value with respect to the X-axis direction. yi is a coordinate value corresponding to the position of the i-th reference point in the captured image I, and indicates an actual measurement value of the coordinate value with respect to the Y-axis direction. Xi is a coordinate value corresponding to the position of the i-th reference point in the captured image I, and indicates a predicted value of the coordinate value with respect to the X-axis direction. Yi is a coordinate value corresponding to the position of the i-th reference point in the captured image I, and indicates a predicted value of the coordinate value with respect to the Y-axis direction. vi is a coordinate value corresponding to the i-th position of the horizontal line in the captured image I, and indicates an actual measurement value of the coordinate value with respect to the Y-axis direction. Vi is a coordinate value corresponding to the i-th position of the horizontal line in the captured image I, and indicates a predicted value of the coordinate value with respect to the Y-axis direction.

That is, the actual measurement values xi and yi are included in the reference point position information, and the actual measurement value vi is included in the horizontal line position information. On the other hand, the predicted values Xi, Yi, Vi are calculated by the external parameter calculation unit 24 while changing the parameter values corresponding to the posture angles ω, φ, κ. That is, the predicted values Xi, Yi, Vi can take different values for each calculation in the iterative calculation. Since the calculation methods of the predicted values Xi and Yi are known, detailed description thereof is omitted. The reference point coordinate information is used for calculating the predicted values Xi and Yi. Since the calculation method of the predicted value Vi is based on the above principle, the description thereof will be omitted.

Thus, by using the horizon position information, the number of reference points used for calculating the posture angles ω, φ, κ can be reduced. In particular, by using two or more pieces of horizontal line position information, the number of reference points used for calculating the posture angles ω, φ, κ can be reduced to one. As a result, even when only one reference point exists within the imaging range of the camera 1, that is, when only one reference point can be prepared in advance, the posture angles ω, φ, κ are calculated. be able to.

Note that the number of input reference point position information is not limited to one, and the number of input horizontal line position information is not limited to two. As described above, the combination of the number of reference point position information and the number of horizontal line position information is based on any combination as long as three or more pieces of position information including one or more reference point position information are input. There may be. However, it is preferable to input two or more pieces of horizontal line position information from the viewpoint of reducing the number of reference points used for calculating the posture angles ω, φ, and κ to one. Further, when increasing the number of input position information, it is preferable not to increase the number of input reference point position information but to increase the number of input horizontal line position information.

That is, by increasing the number of input position information, the calculation accuracy of the posture angles ω, φ, κ can be improved. For example, when the user clicks the position of the reference point or horizontal line in the captured image I using the mouse, the clicked position is converted into the coordinate value in the captured image I. At this time, a so-called “rounding error” occurs. By increasing the number of input position information, the influence of the rounding error on the calculation of the posture angles ω, φ, κ can be reduced.

Further, the horizontal line position information may indicate the position of the horizon in the captured image I instead of the position of the horizontal line in the captured image I. Even in this case, the posture angles ω, φ, κ can be calculated according to the above principle.

As described above, the camera calibration device 100 according to Embodiment 1 includes the reference point position information acquisition unit 21 that acquires the reference point position information indicating the position of the reference point in the captured image I by the camera 1, and the reference point in the real space. A reference point coordinate information acquisition unit 22 that acquires reference point coordinate information indicating coordinate values (that is, absolute coordinate values) corresponding to the position of the horizontal line, and a horizontal line position that acquires horizontal line position information indicating the position of a horizontal line or a horizon in the captured image I Based on the information acquisition unit 23, the position of the reference point in the captured image I, the coordinate value corresponding to the position of the reference point in real space (that is, the absolute coordinate value), and the position of the horizontal line or the horizon in the captured image I And an external parameter calculation unit 24 for calculating the posture angles ω, φ, κ. By using the horizon position information, the number of reference points used for calculating the posture angles ω, φ, κ can be reduced.

In addition, the reference point position information acquisition unit 21 acquires one reference point position information, the reference point coordinate information acquisition unit 22 acquires one reference point coordinate information, and the horizontal line position information acquisition unit 23 2 Get horizontal line position information. By using two or more pieces of horizontal line position information, the number of reference points used for calculating the posture angles ω, φ, κ can be reduced to at least one.

The external parameter calculation unit 24 calculates posture angles ω, φ, κ with respect to the roll direction, the yaw direction, and the pitch direction. That is, the camera calibration apparatus 100 can calculate the triaxial attitude angles ω, φ, and κ using at least one reference point.

Further, the reference point position information, the reference point coordinate information, and the horizontal line position information are input by operating the operation input device 3 on the input screen in a state where the input screen including the captured image I is displayed on the display device 2. Is. Thereby, for example, the input of the reference point position information, the reference point coordinate information, and the horizontal line position information by the UI shown in FIG. 4 can be realized.

Embodiment 2. FIG.
FIG. 7 is a block diagram illustrating a main part of a computer including the camera calibration device according to the second embodiment. With reference to FIG. 7, the camera calibration apparatus 100a of Embodiment 2 is demonstrated. In FIG. 7, the same blocks as those shown in FIG.

The camera calibration device 100 according to the first embodiment uses at least one reference point for calculating the attitude angles ω, φ, and κ. In contrast, the camera calibration apparatus 100a according to the second embodiment uses zero reference points for calculating the posture angles ω, φ, κ, that is, no reference points are required for calculating the posture angles ω, φ, κ. It is what.

The input / output control unit 12 receives input of horizontal line position information. The UI when the horizontal line position information is input is the same as that described with reference to FIG. 4 in the first embodiment. That is, the input screen includes a captured image I obtained by live reproduction. The horizontal line position information acquisition unit 23 acquires the input horizontal line position information when the input / output control unit 12 receives input of horizontal line position information.

The input / output control unit 12 receives an input of an operation indicating that when a single moving body (for example, a ship) is photographed at a predetermined position (for example, the center of the captured image I) in the captured image I It is. This operation is, for example, an operation of pressing the “capture” button displayed in the input screen using the mouse of the operation input device 3. When this operation is input, the input / output control unit 12 temporarily stops the reproduction of the captured image I on the input screen.

Specifically, for example, the input / output control unit 12 captures the captured image I at the timing when this operation is input. The input / output control unit 12 displays the captured still image (hereinafter referred to as “capture image”) instead of the captured image I on the input screen.

The input / output control unit 12 is configured to display the one imaged image I in a state where the reproduction of the imaged image I on the input screen is stopped, that is, in the state where one ship is captured in the center of the imaged image I. The input of information (hereinafter referred to as “moving body area information”) indicating an area corresponding to the ship (hereinafter referred to as “moving body area information”) is received. More specifically, the moving body area is a rectangular area surrounding the one ship, and the moving body area information is information indicating the position and size of the area in the captured image I.

The moving body region information is obtained when the reproduction of the captured image I on the input screen is stopped, that is, when one ship is captured in the center of the captured image I. It is input by operation. This operation is, for example, an operation of designating a moving body region by clicking and dragging using the mouse of the operation input device 3.

The moving body region information acquisition unit 25 acquires the input moving body region information when the input / output control unit 12 receives the input of the moving body region information.

The wireless communication device 4 receives one or more AIS (Automatic Identification System) information corresponding to one or more ships using the antenna 5 under the control of the communication control unit 13. The one or more ships include at least one of a ship that is navigating within a predetermined range including the installation position of the camera 1 or a ship that is anchored within the predetermined range. That is, the one or more ships include at least one ship shown in the central portion of the captured image I.

The wireless communication device 4 includes, for example, an AIS transmitter and receiver. Hereinafter, the AIS information may be referred to as “mobile identification information”.

When a plurality of AIS information is received by the wireless communication device 4, the mobile object identification information selection unit 26 is one that is copied in the central portion of the captured image I among the received plurality of AIS information. One AIS information corresponding to this ship is selected.

That is, the moving body identification information selection unit 26 acquires information indicating the focal length F of the camera 1 from the imaging control unit 11. The moving body identification information selection unit 26 acquires a plurality of AIS information received by the wireless communication device 4 from the communication control unit 13. Each of the plurality of AIS information includes the size S and the absolute coordinate value in the real space of the corresponding ship. The moving body identification information selection unit 26 acquires information indicating the absolute coordinate value of the camera 1 from the external parameter calculation unit 24. The moving body identification information selection unit 26 calculates the distance D between the camera 1 and each ship in the real space based on the absolute coordinate value of the camera 1 and the absolute coordinate value of each ship. The moving body identification information selection unit 26 calculates the size a of each ship in the captured image I when each ship is captured in the captured image I by the following equation (11).

A [pixel] = S [m] × F [pixel] ÷ D [m] (11)

The moving body identification information selection unit 26 compares the calculated size a of each ship with the size of the moving body area indicated by the moving body area information acquired by the moving body area information acquisition unit 25. The moving body identification information selection unit 26 selects one piece of AIS information corresponding to the size a closest to the size of the moving body area.

When the wireless communication device 4 receives one piece of AIS information, the moving body coordinate information acquisition unit 27 acquires the received one piece of AIS information as moving body coordinate information. Moreover, the mobile body coordinate information acquisition part 27 acquires one AIS information selected by the mobile body identification information selection part 26 as mobile body coordinate information, when several AIS information is received by the radio | wireless communication apparatus 4. FIG. To do. That is, the moving body coordinate information includes the absolute coordinate value of one ship that is shown in the center of the captured image I.

The external parameter calculation unit 24 includes the horizontal line position information acquired by the horizontal line position information acquisition unit 23, the moving body region information acquired by the moving body region information acquisition unit 25, and the movement acquired by the moving body coordinate information acquisition unit 27. The posture angles ω, φ, κ of the camera 1 are calculated using the body coordinate information.

That is, the moving body region information acquired by the moving body region information acquisition unit 25 is the position of the moving body region in the captured image I (that is, the imaging) at the timing when one ship is captured in the center of the captured image I. The position of the one ship in the image I). Moreover, the moving body coordinate information acquired by the moving body coordinate information acquisition unit 27 includes the absolute coordinate value of the one ship at the timing. Therefore, the attitude angle ω, φ, κ can be calculated by the same method as that described in the first embodiment by using the one ship instead of one reference point. The external parameter calculation unit 24 calculates the posture angles ω, φ, κ by this method.

After the attitude angles ω, φ, and κ are calculated by the external parameter calculation unit 24, the input / output control unit 12 resumes live reproduction of the captured image I on the input screen. The communication control unit 13 starts continuous reception of AIS information corresponding to one ship that has been captured in the center of the captured image I. The input / output control unit 12 acquires information indicating the absolute coordinate values of the camera 1 and information indicating the posture angles ω, φ, and κ of the camera 1 from the external parameter calculation unit 24. The input / output control unit 12 acquires the AIS information being received by the wireless communication device 4 from the communication control unit 13. The input / output control unit 12 uses these pieces of information to calculate the current position and size of the one ship in the captured image I. The input / output control unit 12 superimposes and displays an image corresponding to the calculated position and size (hereinafter referred to as “superimposed image”) on the captured image I during live reproduction. The superimposed image is, for example, a rectangular frame image. When the superimposed position of the rectangular frame in the captured image I is correct, the ship is displayed in the rectangular frame.

The input / output control unit 12 receives an input of an operation for selecting whether or not the superimposed image is displayed at a correct position in the captured image I. When an operation indicating that the display position of the superimposed image is correct is input, the input / output control unit 12 performs control to output the posture angles ω, φ, κ calculated by the external parameter calculation unit 24. More specifically, the input / output control unit 12 causes the display device 2 to display an image including the calculated posture angles ω, φ, and κ.

The horizontal line position information acquisition unit 23, the external parameter calculation unit 24, the moving body region information acquisition unit 25, the moving body identification information selection unit 26, and the moving body coordinate information acquisition unit 27 constitute a main part of the camera calibration apparatus 100a. . The imaging control unit 11, the input / output control unit 12, the communication control unit 13, and the camera calibration device 100a constitute a main part of the computer 200a.

Since the hardware configuration of the main part of the computer 200a is the same as that described with reference to FIG. 2 in the first embodiment, illustration and description thereof are omitted. That is, the imaging control unit 11, the input / output control unit 12, the communication control unit 13, the horizon position information acquisition unit 23, the external parameter calculation unit 24, the moving body region information acquisition unit 25, the moving body identification information selection unit 26, and the moving body coordinates. Each function of the information acquisition unit 27 may be realized by the processor 31 and the memory 32, or may be realized by the processing circuit 33.

Next, the operation of the computer 200a will be described with reference to the flowchart of FIG. Note that the imaging control unit 11 is executing a process of acquiring image data indicating the captured image I from the camera 1 and outputting the acquired image data to the input / output control unit 12.

First, in step ST21, the input / output control unit 12 starts display of an input screen by the display device 2. The input screen includes a captured image I obtained by live reproduction.

Next, in step ST22, the input / output control unit 12 accepts input of one or more pieces of horizontal line position information according to the operation of the operation input device 3 on the input screen. Next, in step ST23, the horizontal line position information acquisition unit 23 acquires the input horizontal line position information.

Next, in step ST24, the input / output control unit 12 determines whether or not two or more pieces of horizontal line position information have been input. When the number of input horizontal line position information is 1 or less (“NO” in step ST24), the process of the computer 200a returns to step ST22. On the other hand, when the number of input horizontal line position information is 2 or more (step ST24 “YES”), the processing of the computer 200a proceeds to step ST25.

Next, the input / output control unit 12 reproduces the captured image I on the input screen at the timing when one ship is captured in the center of the captured image I in accordance with the operation of the operation input device 3 on the input screen. Is temporarily stopped (step ST25). The processes in steps ST26 to ST33 are executed in a state where reproduction of the captured image I on the input screen is stopped, that is, in a state where one ship is photographed in the center of the captured image I.

Next, in step ST26, the input / output control unit 12 accepts input of moving body region information in accordance with the operation of the operation input device 3 on the input screen. Next, in step ST27, the moving body area information acquisition unit 25 acquires the input moving body area information.

Next, in step ST28, the communication control unit 13 executes control to receive one or more AIS information corresponding to one or more ships. The one or more ships include at least one ship shown in the central portion of the captured image I.

Next, in step ST29, the communication control unit 13 determines whether the number of AIS information received by the wireless communication device 4 is one or more.

When the number of AIS information received by the wireless communication device 4 is plural (step ST29 “YES”), in step ST30, the mobile unit identification information selecting unit 26 selects the plurality of received AIS information. Among them, one AIS information corresponding to one ship shown in the center of the captured image I is selected. Subsequently, in step ST31, the moving body coordinate information acquisition unit 27 acquires the selected one AIS information as moving body coordinate information.

On the other hand, when the number of AIS information received by the wireless communication device 4 is 1 (step ST29 “NO”), in step ST32, the mobile body coordinate information acquisition unit 27 performs the received one AIS information. Information is acquired as moving body coordinate information.

Subsequent to step ST31 or step ST32, in step ST33, the external parameter calculation unit 24, the horizontal line position information acquired by the horizontal line position information acquisition unit 23, the mobile body region information acquired by the mobile body region information acquisition unit 25, And the posture angle ω, φ, κ of the camera 1 is calculated using the moving body coordinate information acquired by the moving body coordinate information acquisition unit 27.

Next, in step ST34, the input / output control unit 12 resumes live reproduction of the captured image I on the input screen. In step ST <b> 35, the communication control unit 13 starts continuous reception of AIS information corresponding to one ship that has been captured in the center of the captured image I.

The input / output control unit 12 acquires information indicating the absolute coordinate value of the camera 1 and information indicating the attitude angles ω, φ, and κ of the camera 1 from the external parameter calculation unit 24. The input / output control unit 12 acquires the AIS information being received by the wireless communication device 4 from the communication control unit 13. The input / output control unit 12 uses these pieces of information to calculate the current position and size of the one ship in the captured image I. The input / output control unit 12 starts superimposed display of an image corresponding to the calculated position and size, that is, a superimposed image (step ST36).

In a state where the superimposed image is superimposed on the captured image I being played live, the input / output control unit 12 inputs an operation for selecting whether or not the superimposed image is displayed at the correct position in the captured image I. Accept (step ST37). When an operation indicating that the display position of the superimposed image is correct is input (step ST37 “YES”), the input / output control unit 12 determines the posture angles ω and φ calculated by the external parameter calculation unit 24 in step ST38. , Κ is output. More specifically, the input / output control unit 12 causes the display device 2 to display an image including the calculated posture angles ω, φ, and κ.

On the other hand, when an operation indicating that the display position of the superimposed image is not correct is input (step ST37 “NO”), the processing of the computer 200a returns to step ST25. Thereby, the generation of the captured image I on the input screen is temporarily stopped at the timing when another ship is captured in the center of the captured image I.

Thus, by using the moving body region information and the moving body coordinate information, a moving body such as a ship can be used instead of the reference point. Thereby, the reference point in the calculation of the posture angles ω, φ, κ can be made unnecessary. As a result, even when there is no reference point in the imaging range of the camera 1, that is, when no reference point can be prepared in advance, the posture angles ω, φ, κ are calculated. be able to.

The horizontal line position information may indicate the position of the horizon in the captured image I instead of the position of the horizontal line in the captured image I. Even in this case, the posture angles ω, φ, κ can be calculated according to the above principle.

Further, the moving body is not limited to a ship, and may be a vehicle or an aircraft. Further, the mobile object identification information may be information including identifiers, sizes, absolute coordinate values, and the like of these mobile objects, and is not limited to AIS information.

Further, the moving body identification information selection unit 26 may be provided outside the camera calibration device 100a. For example, the moving body identification information selection unit 26 may be configured integrally with the communication control unit 13. That is, the horizon line position information acquisition unit 23, the moving body region information acquisition unit 25, the moving body coordinate information acquisition unit 27, and the external parameter calculation unit 24 may constitute a main part of the camera calibration device 100a. .

As described above, the camera calibration device 100a according to the second embodiment includes the horizontal line position information acquisition unit 23 that acquires the horizontal line position information indicating the position of the horizontal line or the horizon in the captured image I by the camera 1, and the predetermined image in the captured image I. A moving body area information acquisition unit 25 that acquires moving body area information indicating an area (that is, a moving body area) corresponding to the moving body in the captured image I when the moving body is captured at the position, and moves to a predetermined position A moving body coordinate information acquisition unit 27 that acquires moving body coordinate information including coordinate values (that is, absolute coordinate values) corresponding to the position of the moving body in the real space when the body is captured; The appearance of the camera 1 based on the position of the horizon, the position of the moving body in the captured image I, and the coordinate values (that is, absolute coordinate values) corresponding to the position of the moving body in real space Angle omega, comprising phi, an external parameter calculation unit 24 for calculating the kappa, the. By using the moving body region information and the moving body coordinate information, a moving body such as a ship can be used instead of the reference point. As a result, the reference point in the calculation of the posture angles ω, φ, κ can be made unnecessary.

Further, the horizontal line position information acquisition unit 23 acquires two pieces of horizontal line position information, the moving body region information acquisition unit 25 acquires one piece of moving body region information, and one moving body coordinate information acquisition unit 27. Get the moving body coordinate information. By using two or more pieces of horizontal line position information, the number of moving bodies used instead of the reference point can be reduced to one.

The external parameter calculation unit 24 calculates posture angles ω, φ, κ with respect to the roll direction, the yaw direction, and the pitch direction. That is, the camera calibration apparatus 100a can calculate the triaxial attitude angles ω, φ, and κ using zero reference points.

The horizontal line position information and the moving body region information are input by an operation of the operation input device 3 on the input screen in a state where the input screen including the captured image I is displayed on the display device 2. The body coordinate information is received by the wireless communication device 4. By using the AIS information as the moving body coordinate information, it is possible to eliminate the need to input the absolute coordinate value of the moving body (more specifically, the ship).

Also, the moving body is a ship. Thereby, AIS information can be used for moving body coordinate information.

In the present invention, within the scope of the invention, any combination of the embodiments, any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

The camera calibration device of the present invention can be applied to a technique for specifying the position of the object in real space according to the position of the object in the captured image when the object is captured in the image captured by the camera. .

1 camera, 2 display device, 3 operation input device, 4 wireless communication device, 5 antenna, 11 imaging control unit, 12 input / output control unit, 13 communication control unit, 21 reference point position information acquisition unit, 22 reference point coordinate information acquisition Unit, 23 horizontal line position information acquisition unit, 24 external parameter calculation unit, 25 mobile unit area information acquisition unit, 26 mobile unit identification information selection unit, 27 mobile unit coordinate information acquisition unit, 31 processor, 32 memory, 33 processing circuit, 100 , 100a Camera calibration device, 200, 200a calculator.

Claims (9)

A reference point position information acquisition unit for acquiring reference point position information indicating the position of the reference point in the image captured by the camera;
A reference point coordinate information acquisition unit for acquiring reference point coordinate information indicating a coordinate value corresponding to the position of the reference point in real space;
A horizontal line position information acquisition unit that acquires horizontal line position information indicating a position of a horizontal line or a horizon in the captured image;
An external parameter for calculating the attitude angle of the camera based on the position of the reference point in the captured image, the coordinate value corresponding to the position of the reference point in the real space, and the position of the horizontal line or the horizon in the captured image A calculation unit;
A camera calibration device comprising: The reference point position information acquisition unit acquires one reference point position information, the reference point coordinate information acquisition unit acquires one reference point coordinate information, and the horizontal line position information acquisition unit 2 The camera calibration apparatus according to claim 1, wherein the horizontal line position information is acquired. 2. The camera calibration apparatus according to claim 1, wherein the external parameter calculation unit calculates the posture angle with respect to a roll direction, a yaw direction, and a pitch direction. The reference point position information, the reference point coordinate information, and the horizontal line position information are input by operating the operation input device on the input screen in a state where the input screen including the captured image is displayed on the display device. The camera calibration apparatus according to claim 1, wherein the camera calibration apparatus is one. A horizontal line position information acquisition unit for acquiring horizontal line position information indicating a position of a horizontal line or a horizon in a captured image by a camera;
A moving body area information acquisition unit that acquires moving body area information indicating an area corresponding to the moving body in the captured image when the moving body is captured at a predetermined position in the captured image;
A moving object coordinate information acquisition unit that acquires moving object coordinate information including coordinate values corresponding to the position of the moving object in real space when the moving object is copied at the predetermined position;
An external parameter that calculates the attitude angle of the camera based on coordinate values corresponding to the position of the horizontal line or the horizon in the captured image, the position of the moving object in the captured image, and the position of the moving object in the real space A calculation unit;
A camera calibration device comprising: The horizontal line position information acquisition unit acquires two pieces of the horizontal line position information, the moving body region information acquisition unit acquires one piece of the moving body region information, and one moving body coordinate information acquisition unit. The camera calibration apparatus according to claim 5, wherein the moving body coordinate information is acquired. 6. The camera calibration apparatus according to claim 5, wherein the external parameter calculation unit calculates the posture angle with respect to a roll direction, a yaw direction, and a pitch direction. The horizontal line position information and the moving body region information are input by operation of the operation input device on the input screen in a state where the input screen including the captured image is displayed on the display device,
The camera calibration apparatus according to claim 5, wherein the moving body coordinate information is received by a wireless communication apparatus. 6. The camera calibration apparatus according to claim 5, wherein the moving body is a ship.

Images