JP7666091B2 - 3D model generating device, 3D model generating method and program - Google Patents

Description

The present invention relates to a three-dimensional model generation device, a three-dimensional model generation method, and a program.

For example, the following Patent Document 1 describes a technology for easily searching for desired image data from multiple image data. The technology described in Patent Document 1 stores multiple image data, to which the location and date of shooting are added as shooting information, in an image exchange server, and searches for desired image data based on the shooting information added to the image data from a terminal device. In addition, the following Patent Document 2 describes a technology for photo mapping multiple images. The technology described in Patent Document 2 performs photo mapping by using meta information added to multiple photo data to perform organic searches, searches for information about buildings within view in real time, and searches for photos in which a specific person is the subject, thereby performing photo mapping.

JP 2004-220420 A JP 2009-176262 A

There is a technology called photogrammetry in which multiple photos are taken of a subject while changing the shooting position, and a 3D model is generated based on the data from the multiple photos. However, conventional photogrammetry generates a 3D model by processing the data from multiple actual photos. Therefore, without actual photo data, it is not possible to generate a 3D model.

The present invention has been made in consideration of the above, and aims to make it possible to generate a three-dimensional model regardless of whether image data is available.

To solve the above-mentioned problems and achieve the object, the three-dimensional model generating device according to the present invention includes an image data acquiring unit that acquires image data of an object including time information, a three-dimensional model generating unit that generates a three-dimensional model based on the image data of the object acquired by the image data acquiring unit, and a three-dimensional model correcting unit that estimates the degree of change of the object at a predetermined time relative to the three-dimensional model and corrects the three-dimensional model.

The three-dimensional model generating method according to the present invention includes the steps of acquiring image data of an object including time information, generating a three-dimensional model based on the acquired image data of the object, and correcting the three-dimensional model by estimating the degree of change of the object at a predetermined time with respect to the three-dimensional model.

The program of the present invention causes a computer operating as a 3D model generating device to execute the steps of acquiring image data of an object including time information, generating a 3D model based on the acquired image data of the object, and correcting the 3D model by estimating the degree of change of the object at a predetermined time with respect to the 3D model.

The present invention has the advantage of being able to generate a three-dimensional model regardless of whether image data is available.

FIG. 1 is a block diagram showing an image acquisition device in a three-dimensional model generating apparatus according to this embodiment. FIG. 2 is a schematic diagram showing a recording format for image data. FIG. 3 is a schematic diagram for explaining position information and orientation information for an object. FIG. 4 is a block diagram showing a three-dimensional model generating device according to this embodiment. FIG. 5 is a flowchart showing a three-dimensional model generating method. FIG. 6 is a block diagram showing a specific configuration of the three-dimensional model generating unit. FIG. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied. FIG. 8 is an explanatory diagram showing the positional relationship between the two images.

The following describes in detail the embodiments of the 3D model generation device, 3D model generation method, and program according to the present invention with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[Image Acquisition Device]
1 is a block diagram showing an image acquisition device in a three-dimensional model generating apparatus according to the present embodiment. The image acquisition device 10 generates format image data (described later) and transmits it to a database 26 in order to cause a three-dimensional model generating apparatus (described later) to generate a three-dimensional model.

As shown in FIG. 1, the image acquisition device 10 includes a camera (imaging device) 11, a GPS receiver (GNSS signal receiver) 12, a direction sensor 13, a time information acquisition unit 21, a position information acquisition unit 22, a direction information acquisition unit 23, a format unit 24, and a memory unit 25.

Camera 11 photographs an object. The image data captured by camera 11 is compressed into a format such as JPEG or PNG to the extent that degradation is not noticeable. Camera 11 sends the captured image data to format unit 24, and also sends it as trigger information to time information acquisition unit 21, position information acquisition unit 22, and orientation information acquisition unit 23.

The GPS receiver 12 receives radio waves from satellites to obtain accurate time information and position information. The GPS receiver 12 sends the time information to the time information acquisition unit 21 and sends the position information to the position information acquisition unit 22.

The time information acquisition unit 21 acquires time information when the camera 11 captured the image data based on time information from the GPS receiving unit 12. The time information acquisition unit 21 sends the time information when the camera 11 captured the image data to the format unit.

The location information acquisition unit 22 acquires location information when the camera 11 captures the image data based on information from the GPS receiving unit 12. The location information acquisition unit 22 sends the location information when the camera 11 captures the image data to the format unit.

The orientation sensor 13 measures the tilt (elevation angle) of the camera 11 using a gyro (angular velocity sensor). The orientation sensor 13 sends the measurement result to the orientation information acquisition unit 23.

The orientation information acquisition unit 23 sends the orientation information when the camera 11 captured the image data to the format unit based on the measurement results of the orientation sensor 13.

The image data captured by the camera 11, the time information of the image data acquired by the time information acquisition unit 21, the position information of the image data acquired by the position information acquisition unit 22, and the orientation information of the image data acquired by the orientation information acquisition unit 23 are input to the format unit 24.

The format unit 24 is set with the number of bytes per parameter, converts the various input information into a specified format, and generates formatted image data. Note that formatted image data is data in which image data, time information of the image data, position information of the image data, and orientation information of the image data are converted into a specified format. The format unit 24 sends the formatted image data to the storage unit 25.

The time information acquisition unit 21, the position information acquisition unit 22, the direction information acquisition unit 23, and the format unit 24 are each configured by an arithmetic circuit such as a CPU (Central Processing Unit).

The storage unit 25 temporarily stores the format image data. The storage unit 25 is an external storage device such as a hard disk drive (HDD) or a memory. The storage unit 25 transmits the format image data to a database 26 connected by a network such as the Internet via a transmission unit (not shown). The format image data temporarily stored in the storage unit 25 is saved on a large scale in the database 26 on the Internet.

Figure 2 is a schematic diagram showing the recording format for image data.

As shown in FIG. 2, the format image data is composed of a fixed-length portion in the first half and a variable-length portion in the second half. The fixed-length portion has time information, location information, and direction information. The time information is year, month, date, and time information, and is composed of 2 bytes of year data, 1 byte of month data, 1 byte of day data, 1 byte of hour data, 1 byte of minute data, and 1 byte of second data. The variable-length portion is composed of image data. The location information is composed of 8 bytes of latitude data and 8 bytes of longitude data. The location information indicates the location where the object was photographed, and information other than latitude and longitude such as address, area name, and building name can be used. Note that the location information may be described as an area divided into a map of a specified size, and the relative coordinates of the horizontal and vertical axes within the area.

The orientation information consists of 8 bytes for the horizontal orientation and 8 bytes for the elevation angle data. The orientation information of the image data acquired by the camera 11 is expressed, for example, using an analog clock, with east being 3 o'clock (90°), south being 6 o'clock (180°), west being 9 o'clock (270°), and north being 12 o'clock (360° (0°)) in a clockwise direction. The elevation angle orientation records the degree of elevation angle when looking up, with the horizontal being 0 degrees and directly above being 90 degrees. The variable-length portion is image information, and the image information is a compressed image file in which the image is compressed into JPG or PNG format.

Figure 3 is a schematic diagram illustrating position information and orientation information for an object.

As shown in Figure 3, map 100 is a division of a city, and map 100 is divided into a number of areas, for example, 100m squares. Figure 3 shows a specific section of map 100 divided into nine areas: area 0010-2324, area 0010-2325, area 0010-2326, area 0010-2455, area 0010-2456, area 0010-2457, area 0010-2586, area 0010-2587, and area 0010-2588.

For example, object T is located in an area with area number 0010-2456. Camera 11 takes images of object T at five positions A1, A2, A3, A4, and A5. Position A1 is the front position relative to object T, position A2 is the left front position relative to object T, position A3 is the left rear position relative to object T, position A4 is the right rear position relative to object T, and position A5 is the right front position relative to object T.

Here, the lines extending from positions A1, A2, A3, A4, and A5 toward the object T are the orientations. The orientation information is horizontal, with the direction directly above the map (due north) being the 12 o'clock direction (0 degrees). For example, the orientation of position A1 is 00 hours, 00 minutes, 00 seconds, and the orientation of position A2 is 02 hours, 30 minutes, 00 seconds. The elevation angle of camera 11, which cannot be shown on map 100, is written in the format as follows: 12 degrees at position A1, 7 degrees at position A2, 10 degrees at position A3, 25 degrees at position A4, and 35 degrees at position A5.

[3D model generation device]
FIG. 4 is a block diagram showing a three-dimensional model generating device according to this embodiment.

As shown in FIG. 4, the three-dimensional model generating device 30 includes an image processing device 31, an input device 32, and a display device 33.

The input device 32 is operated by a user. The input device 32 sends an operation command signal input by the user to the image processing device 31. Here, the operation command signal is the desired location (name and address of the object) for which the user wants to generate a three-dimensional model and the desired time for which the user wants to generate the three-dimensional model. The input device 32 is, for example, a keyboard or a touch panel.

The image processing device 31 is connected to the database 26 via the Internet, for example, via a transmission/reception unit (not shown). The image processing device 31 acquires format image data stored in the database 26 based on an operation command signal sent from the input device 32, and performs image processing on the format image data. That is, the image processing device 31 acquires format image data corresponding to the object stored in the database 26 based on the object sent from the input device 32, and performs image processing on the format image data. The image processing device 31 sends the processed image data to the display device 33.

The display device 33 displays image data to the user. The display device 33 displays the three-dimensional model and image data processed by the image processing device 31. The display device 33 is, for example, a liquid crystal display or a head-mounted display.

The image processing device 31 has an image data acquisition unit 34, a three-dimensional model generation unit 35, and a three-dimensional model correction unit 36. Note that the image data acquisition unit 34, the three-dimensional model generation unit 35, and the three-dimensional model correction unit 36 are configured by, for example, a CPU.

The image data acquisition unit 34 receives an operation command signal input by the user through the input device 32. The image data acquisition unit 34 searches for and acquires a large number of corresponding image data based on this operation command signal. That is, the image data acquisition unit 34 acquires the format image data of the object input from the input device 32 from the format image data stored in the database 26. The image data acquisition unit 34 sends the acquired large number of format image data to the 3D model generation unit 35.

The three-dimensional model generating unit 35 generates a three-dimensional model based on the image data included in the multiple format image data acquired by the image data acquiring unit 34, the time information when the image data included in this format image data was captured, the position information of the image data, and the orientation information of the image data. The three-dimensional model generating unit 35 sends the generated three-dimensional model to the three-dimensional model correcting unit 36.

The three-dimensional model correction unit 36 corrects the three-dimensional model generated by the three-dimensional model generation unit 35 based on the time information input from the input device 32. That is, the three-dimensional model correction unit 36 estimates the time information input from the input device 32, that is, the degree of change of the object at a specified time, and corrects the three-dimensional model. The three-dimensional model correction unit 36 sends the corrected three-dimensional model to the display device 33.

At this time, the three-dimensional model generation unit 35 performs an averaging process on multiple image data with matching time information and orientation information, taking the luminance value of a pixel as the average of the luminance values of the surrounding pixels, and generates a three-dimensional model based on the multiple image data in different orientations that have been averaged. The three-dimensional model correction unit 36 then corrects the three-dimensional model based on the amount of change in the three-dimensional model at different times.

[3D model generation method]
FIG. 5 is a flowchart showing a three-dimensional model generating method.

As shown in FIG. 4 and FIG. 5, in step S11, the user uses the input device 32 to input a desired location for which a three-dimensional model is to be generated as an operation command signal. The desired location is the name or address of an object, etc. In step S12, the user uses the input device 32 to input a desired time for which a three-dimensional model is to be generated as an operation command signal. The desired time is input in terms of the Gregorian calendar year and the year, month, and day. In step S13, the image data acquisition unit 34 searches and acquires a large number of format image data that correspond to the desired location and desired time as search conditions from the database 26. In this case, the image data acquisition unit 34 searches a specified period as the desired time. When, for example, July 1, 2000 is input as the desired time, the image data acquisition unit 34 acquires a plurality of format image data taken during the specified period, that is, January 1, 1995 to July 1, 2005.

In step S14, the three-dimensional model generation unit 35 generates a three-dimensional model based on the image data included in the multiple format image data, the time information of the image data, the position information of the image data, and the orientation information of the image data. Details of the generation of the three-dimensional model in step S14 will be described later. In step S15, the three-dimensional model correction unit 36 estimates the degree of change of the object at a predetermined time for the three-dimensional model and corrects the three-dimensional model. Then, in step S16, the display device 33 displays the corrected three-dimensional model.

That is, first, the image data acquisition unit 34 acquires multiple pieces of format image data from the database 26 via the Internet, thereby acquiring multiple pieces of image data and time information, position information, and orientation information for each piece of image data. The image data acquisition unit 34 acquires multiple pieces of image data photographed at multiple different times and from multiple different positions and orientations relative to the object T.

Next, the three-dimensional model generating unit 35 performs affine transformation with freedom of enlargement, reduction, rotation, left and right keystone correction, etc., on the multiple image data while maintaining the aspect ratio, and calculates the positioning so that the feature points of the images correspond, that is, by matching processing, the difference becomes extremely small. Left and right keystone correction is the principle of perspective, where when a building or the like is photographed from the left side, the left vertical line of the same height is close and therefore appears long, and the right vertical line is farther away and therefore appears short. In the case of normal commemorative photos and other camera photography, the left and right directions can be moved probabilistically, so the number of images taken at many angles increases, but in the up and down directions, the elevation angle changes slightly, and there is not much difference in the up and down of the shooting point. By extracting groups with matching major features, the connections between the images can be understood and are expanded in memory. These processes are photogrammetry techniques, a technology that generates a three-dimensional shape of the object T from images from multiple angles, and are the principles of triangulation. In order to determine whether or not feature points can be obtained, it is preferable that there are two sets of images in the selected images whose fields of view overlap by about 60%.

Then, the 3D model correction unit 36 estimates the degree of change in the 3D model generated by the 3D model generation unit 35 so as to correspond to the desired time, and corrects the 3D model. This makes it possible to generate a 3D model of the object at the specified time desired by the user.

Here, a specific example of the method for generating the three-dimensional model in step S14 will be described. FIG. 6 is a block diagram showing the specific configuration of the three-dimensional model generation unit, FIG. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied, and FIG. 8 is an explanatory diagram showing the positional relationship between the two images.

As shown in FIG. 6, the three-dimensional model generation unit 35 has an epipolar line direction calculator 41, an epipolar line orthogonal direction calculator 42, a search range determiner 43, a corresponding point detector 44, and a distance calculator 45.

The epipolar line direction calculator 41 calculates the direction of an epipolar line connecting corresponding pixel points of the image data for the object T based on the image data acquired by the image data acquisition unit 34. The epipolar line direction calculator 41 sends the calculated direction of the epipolar line to the epipolar line orthogonal direction calculator 42.

The epipolar line orthogonal direction calculator 42 calculates the orthogonal direction that is perpendicular to the epipolar line. The epipolar line orthogonal direction calculator 42 outputs the orthogonal direction to the epipolar line calculated by the epipolar line orthogonal direction calculator 42 to the search range determiner 43.

The search range determiner 43 determines a two-dimensional search range on the screen so as to include multiple pixel points that correspond to the direction of the epipolar line and the direction perpendicular to the epipolar line. The search range determiner 43 outputs the determined two-dimensional search range to the corresponding point detector 44.

The corresponding point detector 44 performs a corresponding point search based on the multiple image data acquired by the image data acquisition unit 34 and the determined two-dimensional search range to obtain a disparity vector. The corresponding point detector 44 sends the obtained disparity vector to the distance calculator 45.

The distance calculator 45 maps the disparity vector onto an epipolar line to obtain the epipolar line direction component of the disparity vector, and calculates the distance to the object T based on the obtained epipolar line direction component. The distance calculator 45 sends the calculated distance to the object T to the 3D model correction unit 36.

The generation of a 3D model is explained in detail below, but here we will explain the case where a 3D model is generated from two pieces of image data.

The image data acquisition unit 34 acquires two pieces of image data from the database 26. For example, as shown in FIG. 3, among the data registered and formatted in area 0010-2456, a second image randomly selected based on one image is used as a set, and a search is made for photographic images that share corresponding points. A three-dimensional point cloud is obtained by triangulating this set of images, and by collecting these, the relative positions and the point cloud image are expanded. In addition, the texture of the part corresponding to this image is associated and managed in memory for later mapping.

First, two sets of image data are obtained for the object T by the camera 11A for the image of the field of view and the camera 11B for the image of the field of view (see FIG. 7 for both). Next, the corresponding point detector 44 searches for corresponding points of the feature points based on the two sets of image data. For example, the corresponding point detector 44 performs correspondence for each pixel and searches for the position where the difference is smallest. Here, as shown in FIG. 6 and FIG. 7, the cameras 11A and 11B, which are assumed to exist simultaneously at two viewpoints, are arranged in a relationship of Yl=Yr so that the optical axes Ol and Or are included on the same X-Z coordinate plane. A parallax vector equivalent to the angle difference for each pixel is calculated using the corresponding points searched for by the corresponding point detector 44.

Since the obtained parallax vector corresponds to the distance from cameras 11A, 11B in the depth direction, distance calculator 45 calculates the distance in proportion to the magnitude of parallax using the law of perspective. If we assume that the photographer's cameras 11A, 11B only move almost horizontally, then by positioning cameras 11A, 11B so that their optical axes Ol, Or are included on the same X-Z coordinate plane, the search for corresponding points can be performed only on the scanning lines, which are epipolar lines Epl, Epr. Distance calculator 45 generates a three-dimensional model of object T using the two image data of object T and the respective distances from cameras 11A, 11B to object T.

On the other hand, when point Ql (Xl, Yl) on the left image corresponds to point Qr (Xr, Yr) on the right image, the disparity vector at point Ql (Xl, Yl) is Vp (Xl-Xr, Yl-Yr). Here, since the two points Ql and Qr are on the same scanning line (epipolar line), Yl = Yr, and the disparity vector is expressed as Vp (Xl-Xr, 0). The epipolar line direction calculator 41 obtains such disparity vectors Vp for all pixel points on the image and creates a group of disparity vectors to obtain information on the depth direction of the image. Incidentally, for sets in which the epipolar line is not horizontal, the height of one of the camera positions may be different (with a low probability). In this case, the epipolar line orthogonal direction calculator 42 searches within a rectangle in the epipolar line direction and in a direction orthogonal to the epipolar line that is approximately the deviation from the horizontal, in a large search range, compared to searching for corresponding points in a large two-dimensional area without considering corresponding point matching, which reduces the amount of calculation for the minimum rectangle and makes it more rational. Then, as shown in FIG. 8, the search range determiner 43 shows the search range when the epipolar line direction search range for the minimum rectangle is a-b = c-d, and the orthogonal direction search range is b-c = d-a. In this case, the search width in the epipolar line direction is ΔE, and the search width in the direction T orthogonal to the epipolar line is ΔT. The smallest non-inclined rectangle ABCD that contains the minimum inclined rectangle abcd is the desired area.

In this way, the 3D model generation unit 35 finds a disparity vector from corresponding points of the feature points of the multiple cameras 11A and 11B under epipolar constraint conditions, obtains depth information for each point, maps the texture on the surface of the 3D shape, and generates a 3D model. This allows the model of the part in the image data used for calculation to reproduce the space seen from the front hemisphere. Furthermore, if there is a part not captured in the image data of the 3D model, and the lines or surfaces of the surrounding texture are extended to connect, the same texture is used to interpolate between them. The 3D model generation unit 35 sends the generated 3D model to the 3D model correction unit 36.

The method for generating the three-dimensional model is not limited to the above, and other methods may be used.

Then, when the 3D model correction unit 36 corrects the 3D model generated by the 3D model generation to a 3D model for a time when no image data exists, it calculates the average value of the image data to estimate the degree of change of the object at a specified time when no image data exists, and corrects the 3D model.

That is, the multiple image data are image data taken during a specified period of time (for example, from January 1, 1995 to July 1, 2005) as the desired time. The multiple image data are grouped by year, and the average value of the color difference signals of the image data in a specified object portion that constitutes the surface of the object that constitutes the target object T is calculated.

The three-dimensional model correction unit 36 determines whether image data at the desired time of the object for which the user wants to generate a three-dimensional model is present in the multiple acquired image data. If multiple image data for, for example, July 1, 2000 exists as the desired time of the object for which the user wants to generate a three-dimensional model, the three-dimensional model correction unit 36 does not correct the three-dimensional model because the three-dimensional model can be generated on July 1, 2000 based on the multiple image data acquired by the three-dimensional model generation unit 35. In other words, if multiple image data for July 1, 2000 exists, the three-dimensional model generation unit 35 calculates the average value of the multiple image data of the object T from the same time, position, and orientation, and generates a three-dimensional model using the color difference signal of the average value of the image data.

On the other hand, if there are no multiple image data for July 1, 2000 as the desired time of the object for which the user wants to generate a 3D model, the 3D model correction unit 36 generates a 3D model for the desired time by correcting the 3D model generated by the 3D model generation unit 35 based on multiple image data that are not the desired time. In this case, the 3D model correction unit 36 corrects the 3D model based on multiple image data before June 30, 2000 and after July 2, 2000. The average value data of the color difference signals of the image data at this time is the average value of the image data in the future direction (before June 30, 2000) and the past direction (after July 2, 2000) of the desired time. The 3D model correction unit 36 corrects the 3D model using the average value of the color difference signals of the image data from different times, the same position, and the same direction. In other words, the 3D model correction unit 36 estimates and synthesizes values obtained by interpolating multiple image data from different times as color difference signals.

In addition, if the image data is in the past (before June 30, 2000) or future (after July 2, 2000) direction, where July 1, 2000, the desired time of the object for which the user wants to generate a 3D model, exists, the average value data of the color difference signals of the image data at this time is the average value of the image data before June 30, 2000, or the average value of the image data after July 2, 2000. The 3D model correction unit 36 estimates and corrects the 3D model in the future or past direction using the average value of the color difference signals of image data from different times, the same position, and the same direction. In other words, the 3D model correction unit 36 estimates and synthesizes values obtained by extrapolating multiple image data from different times as color difference signals. In other words, the 3D model correction unit 36 estimates and synthesizes values obtained by extrapolating image data based on the amount of change in the average value of the color difference signals each year as color difference signals.

In addition, when dividing a day (24 hours) into hourly intervals and using the color difference signals of image data for each hour, the brightness changes depending on the height of the sun depending on the time, and there is a reason to remove anything that affects the color difference signals. For this reason, the 3D model correction unit 36 may obtain weather information from an external source and refer to the weather forecast for the area where the image data was taken, estimate the intensity of the sun's light for each hour of the day, and correct the 3D model.

That is, if there are no multiple image data for July 1, 2000, which is the desired time of the object for which the user wants to generate a 3D model, the 3D model correction unit 36 corrects the 3D model using the average value or amount of change of the image data before June 30, 2000, or the average value or amount of change of the image data after July 2, 2000. At this time, the 3D model correction unit 36 estimates the weather at the desired time, for example, based on the weather before or after July 1, 2000. Then, the 3D model correction unit 36 corrects the 3D model using the average value of the image data with the same weather as the weather on July 1, 2000, from the multiple image data before June 30, 2000 and after July 2, 2000.

The 3D model correction unit 36 may also use image data from a portion of the time when image data exists to perform machine learning using a teacher dataset that takes as input the angle from the camera 11 to the object T at the desired time, July 1, 2000, the weather, and the season depending on the month and day, as well as the photographed image data, and outputs corresponding exemplary photographed image data, thereby synthesizing image data of the object T at a specific time before June 30, 2000 or after July 2, 2000, when no image data exists, using a predicted image.

The image data used by the three-dimensional model correction unit 36 for correction may be image data that predicts the future object T, that is, future prediction image data. Future prediction image data includes space and image data of moving images such as movies, and 3D space and image data created as a virtual future. Image data of the entire configured space can be used, but it also includes partially limited spaces (embedded spaces). It is also possible to perform image processing to generate a composite image by extrapolating or interpolating the space and image data of the pre-development city to new city data, combining the year, month, day, and time information, location information, and orientation information of the space, assuming urban redevelopment or rebuilding of a building. Future prediction image data also includes space and image data created mechanically (such as CAD data) or by artificial intelligence. If the region, building coverage ratio, type of building, etc. are used as input information, spatial image data of a standard structure can be obtained and used.

That is, the database 26 has not only image data of the object T from the past to the present, but also image data predicted for the future (future predicted image data). The future predicted image data is image data that predicts the state of the object T from March 2, 2021 onward, relative to the present day of March 1, 2021. When the user uses the input device 32 to input a future time, for example, March 1, 2031, as the desired time, the image data acquisition unit 34 acquires not only image data before March 1, 2021, but also future predicted image data from March 2, 2021 onward. The three-dimensional model correction unit 36 corrects the three-dimensional model using the image data before March 1, 2021 and the future predicted image data from March 2, 2021 onward.

[Effects of this embodiment]
In this embodiment, the system includes an image data acquisition unit 34 that acquires image data of the object T including time information, a three-dimensional model generation unit 35 that generates a three-dimensional model based on the image data of the object T acquired by the image data acquisition unit 34, and a three-dimensional model correction unit 36 that estimates the degree of change of the object T at a predetermined time relative to the three-dimensional model and corrects the three-dimensional model.

Therefore, the 3D model generation unit 35 generates a 3D model based on multiple image data and time information for each image data, and the 3D model correction unit 36 estimates the degree of change of the object T at a specified time relative to the 3D model and corrects the 3D model. That is, first, a large number of existing image data are acquired, and a 3D model for a specified time is generated from the acquired large number of image data. Next, based on the degree of change (degree of deterioration) of shape, color, etc. estimated from the time in the 3D model for multiple times, the 3D model is corrected to generate a 3D model for a time when no image data exists. As a result, a 3D model can be generated regardless of the presence or absence of image data.

In addition, in this embodiment, the image data of the object T includes position information and orientation information, and the three-dimensional model generation unit 35 generates a three-dimensional model based on the image data of multiple objects T whose time information, position information, and orientation information match. Therefore, it is possible to generate a three-dimensional model of the object T at a specified time with high accuracy.

In addition, in this embodiment, the 3D model correction unit 36 corrects the 3D model based on the amount of change in the 3D model at different times. Therefore, even if image data does not exist for the desired time, a 3D model for the desired time without image data can be generated based on the amount of change in image data other than the desired time.

So far, we have described the 3D model generation device according to the present invention, but it may be implemented in various different forms other than the above-mentioned embodiment.

Each component of the illustrated 3D model generation device is functionally conceptual and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of it may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

The configuration of the three-dimensional model generating device is realized, for example, as software, by a program loaded into memory. In the above embodiment, these functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms, by hardware alone, software alone, or a combination of both.

The above-mentioned components include those that a person skilled in the art would easily imagine and those that are substantially the same. Furthermore, the above-mentioned configurations can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the spirit of the present invention.

10 Image acquisition device 11 Camera (imaging device)
REFERENCE SIGNS LIST 12 GPS receiving unit 13 Orientation sensor 21 Time information acquiring unit 22 Position information acquiring unit 23 Orientation information acquiring unit 24 Format unit 25 Storage unit 26 Database 30 Three-dimensional model generating device 31 Image processing device 32 Input device 33 Display device 34 Image data acquiring unit 35 Three-dimensional model generating unit 36 Three-dimensional model correcting unit T Object

Claims (4)

an image data acquisition unit that acquires image data of an object including time information;
a three-dimensional model generating unit that generates a three-dimensional model based on the image data of the object acquired by the image data acquiring unit;
a three-dimensional model correction unit that estimates a degree of change of the object at a predetermined time with respect to the three-dimensional model and corrects the three-dimensional model;
Equipped with
The image data of the object includes position information and orientation information;
the three-dimensional model generation unit generates a three-dimensional model based on image data of a plurality of objects whose time information, position information, and orientation information match each other;
The three-dimensional model correction unit corrects the three-dimensional model based on weather information of a position where the image data of the object was captured.
3D model generation device. The three-dimensional model correction unit corrects the three-dimensional model based on an amount of change in the three-dimensional model at different times.
The three- dimensional model generating device according to claim 1 . acquiring image data of an object including time information;
generating a three-dimensional model based on the acquired image data of the object;
a step of estimating a degree of change of the object at a predetermined time with respect to the three-dimensional model and correcting the three-dimensional model;
Including,
The image data of the object includes position information and orientation information;
generating a three-dimensional model based on image data of a plurality of objects having the same time information, position information, and orientation information;
correcting the three-dimensional model based on weather information for a location where the image data of the object was captured;
A method for generating 3D models. acquiring image data of an object including time information;
generating a three-dimensional model based on the acquired image data of the object;
a step of estimating a degree of change of the object at a predetermined time with respect to the three-dimensional model and correcting the three-dimensional model;
the image data of the object includes position information and orientation information, generating a three-dimensional model based on image data of a plurality of objects whose time information, position information, and orientation information match, and correcting the three-dimensional model based on weather information at the location where the image data of the object was captured;
A program for causing a computer operating as a three-dimensional model generating device to execute the above.

Images