JP2006304047A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of artificially displaying the photographed scene image as a whole, even if the portion of a scene is covered with an obstacle, such as clouds or fog. <P>SOLUTION: The image processor 1 which displays the entire scene image by interpolating the portion of the scene image covered with the obstacle, when the part of the scene image is covered, has a storage means 5 previously stored with geographical information; a position information read means 3 of reading three-dimensional position information out of input photographic information; a photographic conditions read means 4 of reading photographic conditions; a simulation image forming means 6 of obtaining a scene image viewed from a photography place, based on the three-dimensional position information and photographic conditions; a trimming means 7 of deciding the area covered with the obstacle by comparing outline shapes of the scene information and an ideal scene image with each other and trimming away the decided area from the ideal scene image; and an interpolating means 8 of obtaining an pseudo-scene image which is not covered with the obstacle by interpolating the trimmed ideal scene image for the obstacle. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that obtains a pseudo landscape image obtained by interpolating an obstacle when a part of a photographed landscape image is covered with the obstacle.

  When shooting a landscape such as a distant mountain or mountain using a video camera, still camera, etc., if the part of the landscape is covered with obstacles such as clouds or fog, what kind of scenery is in the covered part There is a possibility that the mountain or mountain you want to check is covered. On the other hand, the technology of navigation systems that can be mounted on the vehicle or carried around has progressed, and was simulated from pre-existing geographic information based on the 3D position information of the shooting location acquired with GPS (Global Positioning System) and altimeter. By displaying the image of the terrain, it is possible to confirm the place where the photographer is and the surrounding area.

Therefore, Patent Document 1 discloses an image processing apparatus capable of integrally displaying a simulated terrain image together with a landscape image obtained by photographing a landscape.
The image processing apparatus described in Patent Document 1 will be described with reference to FIG.
The camera device 27 shown in FIG. 9 is recorded in a photographing unit 24 for photographing a landscape, a recording unit 25 for recording a landscape image photographed by the photographing unit 24, and a landscape and recording unit 25 photographed by the photographing unit 24. Display means 26 for displaying a landscape image.
The image processing device 23 includes a focal length acquisition unit 28 that acquires a focal length used when a landscape image captured by the imaging unit 24 of the camera device 27 is zoomed, and an angle of view from the focal length acquired by the focal length acquisition unit 28. Is stored in the storage unit 30 based on the three-dimensional position information (latitude, longitude, altitude) of the shooting location (viewing angle, azimuth, elevation angle) and the shooting location. Simulated information forming means 31 for simulating an ideal landscape image and place name characters viewed from the camera device 27 from pre-stored geographical information, and the landscape image, the ideal landscape image and place name characters in one screen. And combining means 32 for displaying and outputting to the display means 26.

In this way, the name of the landscape image photographed by the imaging means 24 can be grasped.
JP 2004-356769 A

  However, in the image processing device 23 according to Patent Document 1, when a landscape image obtained by photographing a distant landscape is covered with an obstacle such as a cloud or fog, the covered portion of the landscape is displayed. Because I didn't understand, I couldn't understand the entire landscape image. For this reason, the name and overall shape of the photographed landscape were not known.

  Therefore, the present invention has been made to solve the above problems, and even when a part of a landscape image is covered with an obstacle due to worsening weather conditions, the entire captured landscape image is captured. An object of the present invention is to provide an image processing apparatus that can display in a pseudo manner.

According to a first aspect of the present invention, when a part of a landscape image obtained by photographing a landscape with a photographing unit is covered with an obstacle, the part of the landscape image covered with the obstacle is In the image processing apparatus for interpolating and displaying the entire landscape image, a storage means for storing geographical information on the earth in advance, and a position for detecting geographical three-dimensional position information of the shooting location where the landscape was shot The information detecting means, the photographing condition reading means for reading the photographing conditions set in the photographing means when photographing the landscape, the three-dimensional position information and the photographing conditions are stored in the storage means. Read out the geographical information, simulate image forming means for simulating an ideal landscape image seen from the shooting location where the landscape was shot, and the landscape image and the ideal landscape image shot by the imaging means And trimming the ideal landscape image corresponding to a part of the landscape image covered with the obstacle, based on the landscape image not covered with the obstacle, and the trimming There is provided an image processing apparatus comprising: interpolation means for interpolating the obstacle to the ideal landscape image trimmed by the means to obtain a pseudo landscape image not covered with the obstacle.
The second invention interpolates a part of the landscape image covered with the obstacle when a part of the landscape image obtained by photographing the landscape with the photographing means is covered with the obstacle. In the image processing apparatus for displaying the entire landscape image, storage means for storing geographical information on the earth in advance, and position information reading means for reading the geographical three-dimensional position information of the shooting location where the landscape was shot When shooting the landscape, the shooting condition reading means for reading the shooting conditions set in the imaging means, and the azimuth angle set by the azimuth magnet built in the imaging means is deviated from the geographical azimuth angle. An azimuth angle correction means for correcting an azimuth angle error caused by being generated, and a corrected photographic image for generating a corrected photographic image obtained by correcting the imaging condition based on the corrected azimuth angle The geographical information stored in the storage means is read out using the image forming means, the 3D position information, and the shooting conditions in consideration of the corrected azimuth angle, and the ideal viewed from the shooting location where the landscape is shot Compared with the simulated image forming means for simulating a landscape image, the corrected captured image and the ideal landscape image, and covered with the obstacle based on the landscape image not covered with the obstacle Trimming means for trimming the ideal landscape image corresponding to a part of the landscape image, and a pseudo landscape image that is not covered with the obstacle by interpolating the obstacle to the ideal landscape image trimmed by the trimming means And an interpolation means for obtaining the image processing apparatus.

According to the first aspect of the present invention, storage means in which geographical information on the earth is stored in advance, position information detection means for detecting geographical three-dimensional position information of the shooting location where the scenery is photographed, and the landscape When shooting, shooting condition reading means for reading shooting conditions set in the imaging means, and the geographical information stored in the storage means are read using the three-dimensional position information and the shooting conditions, and the landscape The simulated image forming means for simulating an ideal landscape image viewed from the shooting location where the image was taken is compared with the landscape image captured by the imaging means and the ideal landscape image, and is not covered with the obstacle Trimming means for trimming the ideal landscape image corresponding to a part of the landscape image covered with the obstacle based on the landscape image, and trimming means by the trimming means Interpolating means for interpolating the obstacle to the ideal landscape image thus obtained to obtain a pseudo-landscape image not covered with the obstacle, so that the scenery being photographed is covered with the obstacle The entire portion of the photographed landscape image can be displayed in a pseudo manner.
According to the second aspect of the present invention, the storage means in which geographical information on the earth is stored in advance, the position information reading means for reading the geographical three-dimensional position information of the shooting location where the landscape is shot, and the landscape are shot. The azimuth angle measured and set by the azimuth magnet built in the imaging means is shifted from the geographical azimuth angle. The azimuth angle correcting means that corrects the generated azimuth angle error to obtain a corrected azimuth angle, the three-dimensional position information, the corrected azimuth angle, and the imaging conditions excluding the azimuth angle correction means are stored in the storage means. Simulated image forming means for reading out geographic information and simulating an ideal landscape image viewed from the shooting location where the landscape was shot, and the landscape image and the ideal landscape image shot by the imaging means And trimming the ideal landscape image corresponding to a part of the landscape image covered with the obstacle based on the landscape image not covered with the obstacle, and the trimming means Interpolating means for interpolating the obstacle to the ideal landscape image trimmed by step (a) to obtain a pseudo-landscape image not covered with the obstacle, so that the landscape being photographed is covered with the obstacle. When the portion to be interpolated is interpolated, interpolation processing with less error is performed compared to the configuration according to the first aspect of the present invention, and the entire photographed landscape image can be displayed in a pseudo manner.

The image processing apparatus according to each embodiment of the present invention will be described below with reference to FIGS.
FIG. 1 is a block diagram showing an example of an image processing apparatus according to the first embodiment of the present invention. FIG. 2 is a block diagram showing the trimming means of the first and second embodiments. FIG. 3 is a diagram for explaining an example of processing in the trimming means of the first and second embodiments. (A) shows a landscape image covered with an obstacle, and (B) shows an edge image. (C) shows a blocked edge image. FIG. 4 is a diagram for explaining an example of processing in the trimming means of the first and second embodiments, (A) shows an ideal landscape image, (B) shows an edge image, (C) Indicates a blocked edge image. FIG. 5 is a diagram illustrating an example of processing of the interpolation unit and the synthesis unit according to the first and second embodiments, (A) shows the interpolated pseudo-landscape image, and (B) shows the synthesis of the place name characters. Is shown. FIG. 6 is a block configuration diagram illustrating an example of an image processing apparatus according to the second embodiment. FIG. 7 is a block diagram showing azimuth angle correction means for correcting an azimuth angle error according to the second embodiment. FIG. 8 is a diagram illustrating the influence on the ideal landscape image due to the azimuth angle error according to the second embodiment.

First, the first embodiment will be described.
The image processing apparatus 1 is configured to obtain a whole image by replacing a covered image with a pseudo image when a part of a photographed subject image is covered with an obstacle such as a cloud or a fog. is there.

As shown in FIG. 1, the image processing apparatus 1 replaces a part of a landscape image captured by the camera apparatus 1 with an image to be interpolated when part of the landscape image is covered with an obstacle, and converts the entire landscape image. To get.
The camera device 2 includes a photographing means 3 for photographing a landscape, a video recording means 4 for recording a landscape image photographed by the photographing means 3, and a geographical latitude, longitude, and altitude three-dimensional position indicating a photographing location of the landscape. And a photographing information recording means 5 for recording information and photographing conditions.
The image processing apparatus 1 includes a position information reading unit 6 that reads out the three-dimensional position information recorded in the shooting information recording unit 5 of the camera apparatus 2, and an angle of view, an azimuth angle set in the shooting unit 3 from the imaging information, The photographing condition reading means 7 for reading the photographing condition composed of the elevation angle, the storage means 8 for storing the geographical information on the earth in advance, and the three-dimensional position information and the photographing condition reading means 7 read by the position information reading means 6 Using the obtained shooting conditions, the geographic information of the shooting location is read out from the storage means 8, the simulated image forming means 9 for simulating the ideal landscape image viewed from the shooting location, and the video recording means 4 of the camera device 2. The trimming means 10 that compares the landscape image recorded in the image with the ideal landscape image, extracts an ideal subject image corresponding to a portion where the landscape image is covered with an obstacle, and trims the image. Interpolating means 11 for obtaining a pseudo landscape image by interpolating an ideal landscape image trimmed to a portion where the landscape image is covered with an obstacle, and a place name character for forming a place name character for adding a name to the landscape image from geographic information A forming unit 12; and a synthesizing unit 13 that adds a place name character to the pseudo-landscape image and outputs the image to the display unit 14.

  As shown in FIG. 2, the trimming unit 10 includes an edge detection unit 15 </ b> A that performs edge detection of a contour of a photographed landscape image, an edge detection unit 15 </ b> B that performs edge detection of a contour formed in an ideal landscape image, Blocking units 16A and 16B that divide an edge image obtained by performing edge detection by the edge detection units 15A and 15B into a plurality of blocks and add a block number to each block are provided.

  When the ideal landscape image is compared with the photographed landscape image, and there is a difference in the corresponding contour line between each landscape image, the block of the ideal landscape image corresponding to the photographed landscape image An inter-block difference determining unit 17 that outputs an interpolation block number to be interpolated, and a trimming that outputs an interpolated image obtained by cutting out and trimming the block of the interpolation block number output from the inter-block difference determining unit 17 from the ideal landscape image to the interpolating unit 11 And a processing unit 18.

  Here, the simulated ideal landscape image does not faithfully reproduce the original geographical information up to the shape of the fine terrain that originally exists in the landscape image, so it is covered by the obstacle that is the object of this embodiment. It is determined not to determine even a slight difference in image due to this difference so that only the broken area is extracted. Therefore, in the edge detectors 15A and 15B, a scene image without an obstacle is observed in advance, and determination condition parameters such as a threshold value for binarization of edge detection processing are set so that both edge images have a similar contour shape. Keep it.

  Further, the blocking in the blocking units 16A and 16B is to appropriately determine the area covered by the obstacle, and limits the minimum size of the block. Even if the contour shapes of both edge images are similar, there are errors in each of the three-dimensional position information, shooting conditions, and pre-stored geographic information used when simulating an ideal landscape image. The ideal landscape image has a simulation error in the position, size, inclination of the image, etc., compared with the photographed landscape image. Therefore, when comparing the two edge images, the comparison is made with blocks having an area larger than the area including the simulation error so as not to be determined as an area covered with an obstacle due to the difference between the blocks caused by the error.

  Therefore, it is assumed in advance based on measurement that the simulation error in forming an ideal landscape image is the size of the horizontal direction Dx block and the vertical direction Dy block, and the horizontal direction 2Dx block, the vertical direction 2Dy block or more. The number of divisions is less than the number of divisions by blocks that secure the size of. Here, when the number of divided blocks is made smaller than the above limit, it is set in consideration of the fact that the region other than the obstacle becomes a pseudo-interpolated image.

Next, the operation will be described with reference to FIGS.
First, a landscape is photographed by the photographing means 3 of the camera device 2, a landscape image is recorded in the video recording means 4, and photographing conditions and three-dimensional position information are recorded in the photographing information recording means 5. Thereafter, the three-dimensional position information recorded in the photographing information recording means 5 is read by the position information reading means 6, and the photographing conditions are read by the photographing condition reading means 7. Based on the three-dimensional position information read by the position information reading means 6 and the photographing conditions read by the photographing condition reading means 7, the simulated image forming means 9 designates a geographical range and is designated from the storage means 8. Read out the geographical information of the range and create an ideal landscape image viewed from the shooting location.

Trimming from an ideal landscape image is performed as follows.
Here, the landscape image photographed by the photographing means 3 is a landscape image in which a part shown in FIG. 3A is covered with an obstacle (here, a cloud covering the top of the mountain). Assume that the corresponding ideal landscape image is as shown in FIG.
Edge detection of the landscape image shown in FIG. 3A is performed by the edge detection unit 15A, and the edge image A excluding the obstacle shown in FIG. 3B is generated. The reason why the edge of the obstacle area is not detected is that the change in luminance is so small that it is not determined to be a contour and cannot be represented by a contour line. Next, as shown in FIG. 3C, the edge image A is blocked by the block forming unit 16A in units of a plurality of blocks, and the block edge image A and the block number at the position of each block are determined as the inter-block difference determining unit 17. Output to.

  Similarly, edge detection of the ideal landscape image shown in FIG. 4A is performed by the edge detection unit 15B, and an edge image B shown in FIG. 4B is generated. Next, as shown in FIG. 4 (C), the edge image B is blocked by the block forming unit 16B in units of a plurality of blocks, and the block edge image B and the block number indicating the position of each block are determined as an inter-block difference determining unit. 17 output.

    The contour line between the block image between the block edge images A and B between the landscape image and the ideal landscape image is compared. As shown in FIG. 3C, when the number of blocks in the landscape image is smaller than the number of blocks in the ideal landscape image, it is determined that the landscape image is covered with an obstacle, and the ideal landscape image covered with the obstacle is displayed. The interpolation block number of the part is output to the trimming processing unit 18.

    Further, the interpolation image obtained by trimming the ideal landscape image of the block corresponding to the interpolation block number output to the trimming processing unit 18 and the interpolation block number are output to the interpolation unit 11.

  As shown in FIG. 5A, the synthesizing unit 13 obtains a pseudo landscape image obtained by interpolating an obstacle that covers the landscape image into the interpolated image based on the interpolated image and the interpolation block number output from the interpolating unit 11.

  Further, as shown in FIG. 5B, the place name character formed by the place name character forming means 12 based on the terrain range of the ideal landscape image simulated by the simulated image forming means 9 is combined with the pseudo landscape image to display the display means 14. To display.

  When the landscape image is not covered by an obstacle, the edge images A and B match, so the image displayed on the display unit 14 is not a pseudo-landscape image but a landscape image itself.

  As described above, according to the first embodiment of the present invention, the storage unit 8 for storing geographical information on the earth in advance, and the three-dimensional position information and the photographing condition reading read by the position information reading unit 6 are read. Using the photographing conditions read by the means 7, the geographic information of the photographing location is read from the storage means 8, and the simulated image forming means 9 for simulating the ideal landscape image viewed from the photographing location, and the camera device 2 A trimming unit 10 that compares the landscape image recorded in the video recording unit 4 with the ideal landscape image, extracts an ideal subject image corresponding to a portion where the landscape image is covered with an obstacle, and trims the image; Interpolating means 11 for obtaining a pseudo-landscape image by interpolating an ideal landscape image trimmed to a portion where the landscape image is covered with an obstacle, so that the landscape image is covered with the obstacle. Have Portion is interpolated, covered pseudo pseudo landscape image not can be displayed on the obstacle, it is possible to see the whole scene image.

Next, a second embodiment of the present invention will be described.
The same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.
As shown in FIG. 6, the image processing device 19 in the second embodiment includes an imaging unit 3 between the imaging condition reading unit 7 and the position information reading unit 8 and the simulated image forming unit 9 in the first embodiment. The azimuth angle correction means 20 which correct | amends the azimuth angle error which arises when the azimuth | direction angle set by the azimuth | magnet built in is deviating from the geographical azimuth angle, and produces | generates a correction | amendment azimuth angle is inserted. The others are the same.

As shown in FIG. 7, the azimuth angle correcting means 20 includes an azimuth angle declination table 21 in which declinations corresponding to latitude and longitude are used as a table, and an azimuth angle correction unit 22 that corrects the azimuth angle using the declination from the table. And. In the ideal landscape image shown in FIG. 8A, the center of the landscape image shown in FIG.
The azimuth correcting means 20 matches the ideal landscape image with the landscape image taken at the geographical position based on the error that causes the difference between the direction of the azimuth magnet and the geographical azimuth angle. By doing so, an accurate pseudo-landscape image can be obtained even when a part of the landscape image photographed by the photographing means 3 of the camera device 2 is covered with an obstacle.

  In the first and second embodiments of the present invention, a landscape image photographed by the photographing means 3 of the camera device 2 and photographing information at that time are temporarily recorded and then output to the image processing devices 1 and 19. However, regardless of this, the configuration may be such that the landscape image taken by the photographing means 3 of the camera device 2 and the photographing information at that time are directly output to the image processing devices 1 and 19 during photographing.

  In addition, means having the same function as the video recording means 4 and the photographing information recording means 5 may be installed in the vicinity of the image processing apparatus 1 and the image processing apparatus 19, and these are records recorded in advance by the camera apparatus 2. It is also possible to adopt a configuration in which the media is reproduced and input to the image processing device 1 and the image processing device 19, and the other configuration is the same.

  Furthermore, the video recording means 4 and the photographing information recording means 5 installed in the camera device 2 may be one recording means integrated in the same recording medium. In this case, the reproduced landscape image and photographing information The image processing apparatus 1 and the image processing apparatus 19 can be implemented with the same configuration by providing means for separating them before the input of the image processing apparatus 1, except that the same. is there.

1 is a block configuration diagram illustrating an example of an image processing apparatus according to a first embodiment of the present invention. It is a block diagram which shows the trimming means of 1st and 2nd embodiment. It is a figure explaining the example of the process in the trimming means of 1st and 2nd embodiment, (A) shows the landscape image covered with the obstruction, (B) shows an edge image, (C ) Shows a blocked edge image. It is a figure explaining the example of the process in the trimming means of 1st and 2nd embodiment, (A) shows an ideal scenery image, (B) shows an edge image, (C) shows a block-ized edge An image is shown. It is a figure which shows the example of a process of the interpolation means of 1st and 2nd embodiment, and a synthetic | combination means, (A) shows the pseudo landscape image which was interpolated, (B) shows the synthesis | combination of a place name character. . It is a block block diagram which shows an example of the image processing apparatus which is 2nd Embodiment. It is a block diagram which shows the azimuth angle correction | amendment means which correct | amends the azimuth angle error which is 2nd Embodiment. It is a figure which shows the influence on the ideal scenery image by the azimuth | direction angle error which is 2nd Embodiment. It is a block block diagram which shows the image processing apparatus of a prior art example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 ... Camera apparatus, 3 ... Position information reading means, 4 ... Imaging condition reading means, 5 ... Memory | storage means, 6 ... Simulated image formation means, 7 ... Trimming means, 8 ... Interpolation means, 9 ... Place name character forming unit, 10 ... composing unit, 11 ... display unit, 12A, 12B ... edge detecting unit, 13A, 13B ... blocking unit, 14 ... difference determining unit between blocks, 15 ... trimming processing unit, 16 ... imaging unit, DESCRIPTION OF SYMBOLS 17 ... Image | video recording means, 18 ... Shooting information recording means, 19 ... Image processing apparatus, 20 ... Azimuth angle correction means, 21 ... Azimuth angle deviation table, 22 ... Azimuth angle correction part

Claims (2)

When a part of the landscape image obtained by photographing the landscape by the photographing means is covered with an obstacle, the whole of the landscape image is interpolated by interpolating a part of the landscape image covered with the obstacle. In the image processing apparatus to be displayed,
Storage means in which geographical information on the earth is stored in advance;
Position information detection means for detecting geographical three-dimensional position information of the shooting location where the landscape was shot;
Photographing condition reading means for reading photographing conditions set in the imaging means when photographing the landscape;
Simulated image forming means for reading out the geographical information stored in the storage means using the three-dimensional position information and the photographing conditions, and simulating an ideal landscape image viewed from the photographing location where the landscape is photographed. ,
Comparing the landscape image captured by the imaging means with the ideal landscape image and corresponding to a part of the landscape image covered with the obstacle based on the landscape image not covered with the obstacle Trimming means for trimming the ideal landscape image;
Interpolating means for interpolating the obstacle to the ideal landscape image trimmed by the trimming means to obtain a pseudo landscape image not covered with the obstacle;
An image processing apparatus comprising: When a part of the landscape image obtained by photographing the landscape by the photographing means is covered with an obstacle, the whole of the landscape image is interpolated by interpolating a part of the landscape image covered with the obstacle. In the image processing apparatus to be displayed,
Storage means in which geographical information on the earth is stored in advance;
Position information reading means for reading geographical three-dimensional position information of the shooting location where the landscape was shot;
Photographing condition reading means for reading photographing conditions set in the imaging means when photographing the landscape;
An azimuth correction unit that corrects an azimuth error caused by a deviation of the azimuth angle set by the azimuth magnet built in the imaging unit from a geographical azimuth angle, and generates a corrected azimuth angle;
A corrected captured image generating means for generating a corrected captured image obtained by correcting the imaging condition based on the corrected azimuth angle;
The geographical information stored in the storage means is read using the three-dimensional position information and the photographing condition considering the corrected azimuth angle, and an ideal landscape image viewed from the photographing location where the landscape is photographed is simulated. Simulated image forming means for
Comparing the corrected captured image with the ideal landscape image, and based on the landscape image not covered with the obstacle, the ideal landscape image corresponding to a part of the landscape image covered with the obstacle Trimming means for trimming,
Interpolating means for interpolating the obstacle to the ideal landscape image trimmed by the trimming means to obtain a pseudo landscape image not covered with the obstacle;
An image processing apparatus comprising:

Images