JP6526605B2 - Virtual camera image generating device - Google Patents

Description

  According to the present invention, the three-dimensional object is imaged from a virtual viewpoint arbitrarily set by the user or the like based on an object image of the three-dimensional object extracted from each of images obtained by simultaneously imaging the three-dimensional object by a plurality of cameras. The present invention relates to an apparatus for generating an image.

  In recent years, based on images obtained by capturing a predetermined three-dimensional object (for example, a person) from various viewpoints by a plurality of cameras, the three-dimensional object is captured from a virtual viewpoint (hereinafter, virtual camera) arbitrarily set by a user or the like. Techniques for generating virtual images of when (hereinafter, virtual camera images) are used in various fields.

  For example, in Patent Document 1 below, the three-dimensional shape of the three-dimensional object is modeled as a set of voxels based on an object image of a predetermined three-dimensional object extracted from images simultaneously captured from a plurality of cameras, and the voxels are projected Thus, a method of generating a virtual camera image is described. In the related art, the visual volume intersection method is used to model the three-dimensional shape of a three-dimensional object. That is, a cone (view volume) obtained by back-projecting the object image for each camera is determined, a product set of view volumes of a plurality of cameras is generated, and this is made a three-dimensional shape of a three-dimensional object.

Japanese Patent Laid-Open No. 2000-306117

  In the conventional method of generating a virtual camera image using the view volume intersection method, the resolution of the image is increased by the reduction of the voxel size. Here, the computational complexity in the visual volume intersection method is proportional to the number of voxels. The number of voxels basically increases in proportion to the cube of the inverse of the voxel size and also in proportion to the volume of the space or solid object to be calculated. That is, as the voxels are made smaller to increase the resolution, the amount of calculation increases in response to the sudden increase in the number of voxels. Therefore, in the generation of a virtual camera image in a wide space, there is a problem that it may be substantially difficult to realize a sufficient image resolution due to the performance limit of a computer that performs image generation processing. On the other hand, if the voxel size is increased to reduce the amount of calculation, the resolution of the object image displayed in the virtual camera image is reduced.

  The present invention has been made to solve the above-mentioned problems, and it is possible to make it difficult to be restricted by the performance limit of a computer and to obtain a virtual camera image of sufficient resolution even in a wide space. Intended to be provided.

  (1) The virtual camera image generation apparatus according to the present invention extracts an object image of the solid object from each of the real images obtained by simultaneously imaging the solid object from different viewpoints by a plurality of real cameras, and based on the object image An image generating apparatus for generating a virtual camera image obtained by imaging the three-dimensional object with a virtual camera, storing real camera information representing camera parameters of the real camera and virtual camera information representing camera parameters of the virtual camera A storage unit, projection line generation means for obtaining a virtual projection line corresponding to each pixel of the virtual camera image using the virtual camera information, an attention point is set on the virtual projection line, and the real camera information is used The projection position of the target point in each of the real images is determined, and the target point at which the projection position is within the With an object corresponding point detecting means for, and a object image generating means for generating an image of the three-dimensional object in the virtual camera image by the pixels corresponding to the virtual projection line the object corresponding point is detected.

  (2) In the virtual camera image generation device according to (1), the object corresponding point detection unit sets the attention point on each virtual projection line at each reference distance interval smaller than the assumed size of the three-dimensional object. It can be configured.

  (3) In the virtual camera image generating device according to the above (1) and (2), the object corresponding point detecting means sets the target points for each reference distance interval on the virtual projection line to set the object corresponding points. And setting the target points sequentially from the reference position toward the virtual camera at an interval smaller than the reference distance interval, the target points not being the object corresponding points Performing a surface position search process for searching for and estimating the surface position of the three-dimensional object on the virtual projection line, and the object image generation unit determines the virtual image based on the surface position on the virtual projection line The pixel value of the pixel corresponding to the projection line can be determined.

  (4) In the virtual camera image generating device according to (3), the object corresponding point detecting means further determines in the direction opposite to the surface position from the reference position on the virtual projection line from the reference distance interval. The back surface position search process of estimating the back surface position of the three-dimensional object on the virtual projection line is performed by sequentially setting the target points at small intervals and searching for the target points that are not the object corresponding points. The generation means may be configured to determine the pixel value of the pixel corresponding to the virtual projection line according to the object size in the gaze direction estimated from the front surface position and the back surface position on the virtual projection line. .

  (5) In the virtual camera image generating device according to the above (2) to (4), the object corresponding point detecting means is arranged around the object corresponding point detected in another virtual projection line among the virtual projection lines. The target point may be set at an interval smaller than the reference distance interval in a portion included in the predetermined near space.

  (6) In the virtual camera image generating device according to (1) to (5), the storage unit stores a three-dimensional space range in which the three-dimensional object may exist, and the object corresponding point detection unit The target point can be set only within the range.

  According to the present invention, it is easy to obtain a virtual camera image of sufficient resolution even in a wide space.

FIG. 1 is a schematic block diagram of an image monitoring system according to an embodiment of the present invention. It is a schematic diagram explaining the process of the projection line production | generation means in the image production | generation apparatus which concerns on embodiment of this invention. It is a flowchart of the outline of restoration processing of the attention object image in the image generation device concerning the embodiment of the present invention. It is a schematic diagram which shows the example of the determination area in the case where determination area | region information is represented by the bounding box. It is a schematic diagram explaining the surface position search process by the object corresponding point detection means in the image generation apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention (hereinafter referred to as embodiments) will be described based on the drawings.

  FIG. 1 is a schematic block diagram of an image monitoring system 1. The image monitoring system 1 is a mechanism for collectively displaying images (hereinafter, real images) captured by a plurality of monitoring cameras installed on a monitored object and supporting visual confirmation of images by a monitoring worker. . The image monitoring system 1 includes an imaging device 2, an input device 3, an image generation device 4, and an output device 5. Among these, the image generation device 4 constitutes the main part of the virtual camera image generation device according to the present invention.

  The property to be monitored is, for example, a monitoring area set inside or outside a building, and examples of a monitoring area for outdoor monitoring include a site on the outer periphery of a building, a garden, and an approach. Also, regarding indoor monitoring, there are examples such as entrance halls and lobbies of apartments, office buildings, hotels and the like.

  The imaging device 2 is a surveillance camera installed in a surveillance area, and functions as a real camera in the present invention. A plurality of imaging devices 2 are installed so as to be able to capture an object of interest (for example, a person) to be monitored from different viewpoints. The imaging device 2 is connected to the image generation device 4, and outputs the captured image to the image generation device 4.

  The input device 3 is a mouse or a keyboard operated by a surveillance worker to control the operation of the image surveillance system 1. The input device 3 is connected to the image generation device 4, and various information is input to the image generation device 4 from the input device 3.

  The image generation device 4 is a computer including a central processing unit (CPU), a graphics processing unit (GPU), various memories, and the like. The image generation device 4 uses an actual image input from the imaging device 2 to use The image (hereinafter referred to as the object image of interest) is restored. The image generation device 4 includes a storage unit 41, a target object image extraction unit 42, a target object image restoration unit 43, and an image output unit 44.

  The storage unit 41 stores a computer program for controlling the image generation device 4 and various information. The various information stored in the storage unit 41 is updated by the input device 3 and the target object image extraction unit 42, and is referred to by the target object image restoration unit 43. The various information includes an extracted object image 411, virtual camera information 412, determination area information 413, and real camera information 414.

  The extracted object image 411 is an attention object image extracted from the real image, and is updated by the attention object image extraction unit 42.

  The virtual camera information 412 is a camera parameter that defines the field of view of a virtual camera that is a camera virtually disposed in a three-dimensional space, and defines a camera model when the image generation device 4 restores an object image of interest. That is, the image generation device 4 restores the target object image as an image captured by the virtual camera. Specifically, virtual camera information 412 includes information on visual field conversion, projection conversion, and the size of the image to be generated. The information on the visual field conversion includes the position (center coordinates of the lens or the viewpoint) and the attitude (the direction of the lens optical axis or the direction of the line of sight) of the virtual camera, and can be updated any time by the operation of the input device 3 by the monitoring worker. The information on the projection conversion includes a group of parameters for modeling the projection characteristics of the lens, such as focal length and distortion aberration coefficient, and the information on the size of the image includes the number of pixels forming the image. The information on the conversion and the size of the image are given predetermined values.

  The determination area information 413 is information that defines a space range in which the attention point is set in processing of determining whether the attention point in the three-dimensional space to be monitored corresponds to an object. The spatial range is an area where the restoration determination process of the object of interest described later is performed. In the present embodiment, a predetermined value is given to the determination area information 413 in advance. Here, the attention point is a point to be determined.

  In addition, as a method of expressing the determination area information 413, for example, a method of expressing in a rectangular parallelepiped (bounding box) including an area in which a target object can exist in a three-dimensional space to be monitored can be considered.

  The real camera information 414 is a camera parameter that defines the field of view of each imaging device 2, and is a group of parameters for modeling the position and orientation of each imaging device 2 and the projection characteristics of the lens mounted on the imaging device 2. Is included. For the actual camera information 414, a value obtained in advance by measurement or the like for each imaging device 2 is used.

  The object-of-interest image extraction unit 42, the object-of-interest image restoration unit 43, and the image output unit 44 are realized by the processor of the computer configuring the image generation device 4 reading out a program from the storage unit 41 and executing it.

  The object-of-interest image extraction unit 42 acquires a real image from the imaging device 2, extracts the object-of-interest image from the real image, and updates the extracted object image 411 stored in the storage unit 41.

  Many methods have been proposed in the past for extracting the object of interest from the actual image, and if the object of interest is a moving object, there is a method of background difference that takes the difference with the background image captured when there is no moving object There is.

  The attention object image restoration unit 43 refers to various information stored in the storage unit 41, and restores the attention object image from an arbitrary viewpoint. The object-of-interest image restoration unit 43 includes a projection line generation unit 431, an object corresponding point detection unit 432, and an object image generation unit 433.

  The projection line generation unit 431 obtains a virtual projection line corresponding to each pixel constituting the virtual camera image based on the virtual camera information 412. The virtual camera image is an image generated by the image generation device 4 as an image captured by the virtual camera. FIG. 2 is a schematic view for explaining the processing of the projection line generation unit 431, and is a perspective view showing the geometrical relationship between the virtual viewpoint 21 of the virtual camera, the virtual camera image 22 and the virtual projection line 23. The virtual projection line 23 is generated for each pixel 22 p of the virtual camera image 22 and is a half straight line extending from the virtual viewpoint 21 to the infinite point through the pixel 22 p.

  The object corresponding point detection means 432 sets the point of interest on the virtual projection line, determines the projection position in each real image of the point of interest based on the real camera information 414, and the projection position in all real images is from the real image A point of interest within the extracted extracted object image 411 is determined to be a point located within the object of interest in the three-dimensional space, and is detected as an object corresponding point.

  The object corresponding point detection unit 432 sets the target point only within the space range specified by the determination area information 413.

  The object image generation unit 433 generates an image of a three-dimensional object in the virtual camera image by pixels corresponding to the virtual projection line in which the object corresponding point is detected, and thereby the object image of interest is restored. The object image generation unit 433 can set the pixel values of all the pixels constituting the target object image to the same value, or can set different pixel values for each pixel. For example, the object image generation unit 433 may perform texture mapping on the restored target object image. For example, in the texture mapping process, the surface position of a three-dimensional object on a virtual projection line is determined by a method to be described later, the surface position of the three-dimensional object is projected onto a real image, and the pixel value of the projection position is restored. It will be a value. Further, the object image generation unit 433 determines the pixel value of the pixel corresponding to the virtual projection line based on the surface position, and for the pixel value, according to the size of the three-dimensional object in the direction of the virtual projection line. You can also make adjustments. For example, the hue of the pixel can be changed according to the distance of the surface position from the virtual viewpoint, and the saturation can be set larger as the object size in the viewing direction is larger.

  The image output unit 44 generates an image obtained by combining the object image generated by the target object image restoration unit 43 with the background image of the monitoring area viewed from the virtual viewpoint. The background image can be configured to be separately created based on the virtual camera information 412 and read in advance and stored in the storage unit 41 for use.

  The output device 5 is a display, a projector, or the like that displays an image generated by the image generation device 4. Specifically, the output device 5 displays the image generated by the image output unit 44.

  Next, the operation of the image monitoring system 1 will be described.

  FIG. 3 is a flow chart outlining the process of restoring the object of interest. The process from the start to the end of the flow shown in FIG. 3 corresponds to the procedure necessary to generate an image of one frame. The process of extracting the object of interest from the actual image and storing the extracted object image 411 in the storage unit 41 is sequentially executed each time a new actual image is input from the imaging device 2 and stored in the storage unit 41. The existing virtual camera information 412 is updated as needed by the input device 3 operated by the surveillance worker. Further, when the object-of-interest image restoration unit 43 completes the process of restoring the object-of-interest image shown in FIG. 3, subsequently, the image output unit 44 executes processing for generating a virtual camera image based on the restoration result of the object-of-interest image. Ru.

  The target object image restoration unit 43 repeats the processing from step S2 to step S6 for each pixel constituting the virtual camera image. The virtual camera information 412 stored in the storage unit 41 is referred to for the size (number of pixels) of the virtual camera image.

  The object-of-interest image restoration unit 43 functions as a projection line generation unit 431, and based on the coordinates of the pixel and the virtual camera information 412, a virtual projection line starting from the virtual viewpoint and passing through the pixel selected as the processing target in step S1. Calculation (step S2).

  After obtaining the virtual projection line, the target object image restoration unit 43 functions as an object corresponding point detection unit 432, and calculates a determination section based on the determination area information 413 stored in the storage unit 41 (step S3). . The determination section is a section for setting an attention point in the virtual projection line, and is a portion of the virtual projection line existing in the spatial range defined by the determination area information 413. FIG. 4 is a schematic view showing an example of the determination section in the case where the determination area information 413 is expressed by the bounding box, and in the determination section, the virtual projection line 23 extending from the virtual viewpoint 21 is a boundary surface of the bounding box 25. It becomes a section which makes two points Pn and Pf which intersect be both ends.

  The object corresponding point detection means 432 sequentially sets an attention point in the determination section, and the region of presence of the object of interest in the three-dimensional space estimated from the attention object image of the real image and the coordinates of each attention point Intersection determination between the object of interest and the virtual projection line is performed by obtaining the inclusion relation with the point coordinates) (steps S4 to S6).

  Various methods can be considered as how to set the focus point in the determination section. For example, in the example of FIG. 4, the point of interest may be set to an initial value coordinate as the point Pn, and may be sequentially moved along the virtual projection line 23 in the direction of the point Pf.

  Here, the object corresponding point detection unit 432 determines that the virtual projection line intersects the target object when any target point set on the virtual projection line is included in the existing area of the target object in the three-dimensional space. . Therefore, it is preferable to set the reference distance to a value smaller than the depth of the object of interest in order to avoid failing to detect the virtual projection line intersecting the object of interest. Specifically, the object corresponding point detection unit 432 sets an attention point on each virtual projection line at each reference distance interval smaller than an assumed size which is a previously assumed depth of the attention object.

  When the object corresponding point detecting means 432 updates the point of interest coordinates (step S4), the object corresponding point detecting means 432 checks whether or not the point of interest is present in the determination section obtained in step S3 (step S5). When the attention point is present in the determination section (in the case of “YES” in S5), the object corresponding point detection means 432 determines whether the attention object includes the attention point coordinates (step S6). On the other hand, when the attention point does not exist in the determination section (in the case of “NO” in S5), the processing for the pixel to be processed is finished, and the process returns to step S1 to shift to the processing of the next pixel.

  When it is determined in step S6 that the point of interest coordinates set in step S4 are located within the object of interest (in the case of “YES” in S6), the object corresponding point detection unit 432 detects, for example, the pixel to be processed Then, the process proceeds to step S1 to shift to the process of the next pixel. On the other hand, when it is determined that the focus point coordinates are not within the focus object (in the case of “NO” in S6), the object corresponding point detection means 432 proceeds to step S4 and updates the focus point coordinates.

  In the process of determining whether the object of interest includes the coordinates of the point of interest, the object corresponding point detection means 432 projects the point of interest onto the image plane of each imaging device 2 using the actual camera information 414, and the projection coordinates It is determined whether it is included in the image of the object of interest by the imaging device 2. And about all the imaging devices 2, when a projection coordinate is in an attention object image, it determines with an attention point existing in the area | region where the attention object in three-dimensional space exists, and determines the said attention point as an object corresponding point. Do. That is, the virtual projection line 23 having the object corresponding point intersects the object of interest.

  In the case of texture mapping an actual image on a target object image of a virtual camera image, it is necessary to accurately calculate the distance between the target object and the virtual viewpoint. Therefore, the object corresponding point detection unit 432 can have a function of obtaining an object corresponding point that can be seen as the surface position of the target object on the virtual projection line.

  The function can also be realized by reducing the reference distance interval, but the object corresponding point detection means 432 performs surface position search processing for estimating the surface position with a smaller amount of calculation. Specifically, the object corresponding point detecting means 432 sets an attention point for each reference distance interval on the virtual projection line as a reference position searching process, searches for an object corresponding point, and sets the object corresponding point as a reference position. The processing to determine is performed, and then the surface position searching processing is performed. In the surface position search process, attention points are sequentially set at intervals smaller than the reference distance interval from the reference position toward the virtual camera, and attention points that are not object correspondence points are searched, and the attention object on the virtual projection line Estimate the surface position of For example, when it is determined that the n-th target point in the surface position search process is an object corresponding point and the n + 1th target point is not an object corresponding point, the surface position is between the target points. Become. Therefore, the object corresponding point detection unit 432 can, for example, approximate the coordinates of the n-th attention point as the surface position.

  FIG. 5 is a schematic view for explaining the surface position searching process. In the reference position search process, the attention point coordinates at the reference distance interval α set according to the size of the attention object from the end point Pn on the near side of the judgment section to the direction opposite to the virtual viewpoint 21 (the direction indicated by the arrow 26). The “(●)” mark is moved and updated to determine whether the object of interest 27 includes the coordinates of the point of interest. Then, an attention point Pa first determined to be included in the attention object 27 is obtained, and this is used as a reference position. Next, in the surface position search process, the point of interest coordinates ("o" mark) are moved and updated at an interval β smaller than the reference distance interval α in the direction from the reference position Pa toward the virtual viewpoint 21 (the direction indicated by the arrow 28). Then, it is determined whether the object of interest 27 includes the coordinates of the point of interest. Then, the attention point is updated until the first attention point Pc not included in the attention object appears. When the point of interest Pc is obtained, it is known that the surface position of the object of interest exists between the point of interest Pc and the point of interest Pb immediately before it, so the update of the point of interest is stopped.

  By this processing, a point Pb closer to the surface than the reference position Pa searched at a relatively large interval α can be obtained, and the distance between the virtual viewpoint and the object of interest can be determined more precisely. In this process, the number of target points to be set is likely to be smaller than in the method of sequentially setting the target points at a small interval β from the point Pn and searching for the surface position, and the calculation amount can be suppressed.

  In addition, the object corresponding point detection unit 432 may perform back surface position search processing for estimating the back surface position of the target object on the virtual projection line. In the back surface position searching process, attention points are sequentially set at an interval β ′ smaller than the reference distance interval α from the reference position on the virtual projection line obtained by the above-described reference position searching process to the direction opposite to the surface position. An attention point which is not an object correspondence point is searched for, and a back surface position of the attention object on the virtual projection line is estimated. Note that β ′ can be set to either the same value as β or a different value.

  The front surface position and the back surface position on the virtual projection line calculated by the object corresponding point detection unit 432 can be used in the process of determining the pixel value of the pixel corresponding to the virtual projection line. For example, the object image of interest can be shaded according to the front surface position or the back surface position to give a three-dimensional effect.

  Further, for example, as texture mapping using a surface position on a virtual projection line, the object image generation unit 433 may, for example, perform the surface position on the imaging device 2 having a viewpoint capable of photographing the surface of the object of interest corresponding to the surface position. It is possible to project the point of and set the pixel value of the actual image at the projection position as the pixel value corresponding to the virtual projection line.

  Further, the object corresponding point detection means 432 calculates both the front surface position and the back surface position, and the object image generation means 433 calculates the object size in the gaze direction estimated from the front surface position and the back surface position on the virtual projection line In accordance with the thickness of (1), it is also possible to perform a process of determining the pixel value of the pixel corresponding to the virtual projection line. For example, if the thickness of an object in a certain virtual projection line is equal to or less than the assumed size of the target object, the object captured in the virtual projection line is not displayed in the virtual camera image as not being the target object. . Thus, for example, when a person is the object of interest, it is possible to remove noise components such as insects, leaves of trees, ornaments, etc. floating in the space, and generate a virtual camera image consisting of a required human image.

  Furthermore, as another function of the object corresponding point detecting means 432, a reference distance interval α in a portion included in a predetermined vicinity space around an object corresponding point detected in another virtual projection line among virtual projection lines. A near space search process can also be provided in which the focus points are set at an interval γ smaller than that. For example, when the object corresponding point detection unit 432 detects an object corresponding point on a virtual projection line corresponding to a certain pixel of the virtual camera image, it sets a predetermined near space around the object corresponding point. The near space has, for example, a range including a virtual projection line of eight pixels in the vicinity of the pixel in the field of view in the virtual viewpoint, and of the virtual projection lines, the distance from the virtual viewpoint is [R−δ1, R + δ2]. It can be set to include a certain part. Here, R is the distance from the virtual viewpoint of the detected object corresponding point. Further, δ1 and δ2 are values set larger than γ, and δ1 may be δ2. γ can be set separately in the direction along the virtual projection line and in the direction orthogonal to it. For the direction along the virtual projection line, γ is defined as a fixed distance, and in the orthogonal direction, γ is adjacent It can be defined by the angle between virtual projection lines.

  Generally, in the surrounding space in which an object corresponding point is detected, there is a high possibility that there are other object corresponding points constituting the object of interest. Therefore, in the near space search processing described above, the attention point is set at a distance interval γ smaller than the reference distance interval α in the near space, and the object corresponding point is searched for, thereby reducing the size of the portion constituting the attention object Even for a part, it is possible to obtain an object corresponding point, and a precise object image in a virtual camera image can be obtained.

  A new near space may be set based on the object corresponding point newly detected in the near space search process, and the near space search process may be further performed. At that time, a new search is performed in the part excluding the space searched by the processing up to that point.

  In the reference position searching process, the points of interest are sequentially set from the point Pf toward the point Pn contrary to the process of setting the points of interest sequentially from the point Pn to the point Pf described with reference to FIG. You may Further, the setting of the attention point is not limited to the method of sequentially shifting in one direction at fixed intervals. For example, a point of interest P1 is first set at a position that divides the determination section into two, and then points of interest P2 and P3 are set at positions that divide each of two sections generated by division by P1 in the determination section. This procedure may be repeated to sequentially set an attention point that divides a section into two. In addition, this process can be completed at the stage where the interval of the attention points becomes equal to or less than the size of the attention object.

  The resolution of the virtual camera image increases with the number of pixels. That is, the resolution is determined according to the number of virtual projection lines set within the angle of view of the virtual camera, and in the case where the angle of view is the same, the larger the density of the virtual projection line, the higher the resolution. In the present invention, the pixel value is calculated by processing for each virtual projection line, so the processing is a calculation amount according to the density of the virtual projection line. On the other hand, when generating a virtual camera image by projecting a three-dimensional object modeled by voxels, when trying to obtain a resolution according to the density of the virtual projection line, the density of voxels in the direction orthogonal to the virtual projection line is basically It should be set higher than the density of the virtual projection line. In this respect, the process of generating a virtual camera image according to the present invention can reduce the amount of calculation as compared with the method using voxels.

  In addition, in the case of projecting a three-dimensional object modeled by voxels and generating a virtual camera image with respect to the direction along the virtual projection line, a process of basically calculating the projection position in the virtual camera image for all voxels Do. On the other hand, in the present invention, processing is performed to set an attention point discretely on a virtual projection line and search for an object corresponding point. Since the processing for each virtual projection line can basically be finished at the point where it is detected as an object corresponding point at any point of interest, the amount of calculation is generally larger than the method using voxels. It can be expected to decrease.

  In the above embodiment, as the space range for setting the point of interest, the area where the object of interest may exist in the three-dimensional space is set in advance regardless of where the object of interest is actually located in the three-dimensional space. And stored in the storage unit 41 as the determination area information 413. On the other hand, the spatial range in which the attention point is set can also be determined based on the extracted object image 411. For example, from the extracted object images 411 of the plurality of imaging devices 2, a region that is considered to be a target object in the three-dimensional space is estimated, and the object corresponding point detection unit 432 performs the processing within the region. As a result, the amount of calculation of the search process for object corresponding points can be further reduced.

  Further, in the above embodiment, the object corresponding point detection means 432 projects the attention point on the image plane (real image) of all the imaging devices 2 and the projection position is included in the attention object image in all real images. Although it is determined whether or not the target point is an object corresponding point by determining whether or not it is selected, the present invention is not limited to this. That is, it may be determined that the target point included in the target object image in the plurality of (at least two or more) real images instead of “all” is the object corresponding point. Generally, as the number of imaging devices 2 in which the projection position is included in the target object image of the actual image is large, the possibility that the target point corresponding to the projection position is the object corresponding point is high. Therefore, in the case of such another embodiment, the restoration accuracy of the object image of interest in the virtual camera field image generally lowers than in the embodiment in which “all” is the determination condition. Therefore, in such another embodiment, when the object image generation unit 433 performs the texture mapping process, the number of imaging devices 2 in which the projection position is included in the target object image of the real image is large It is preferable to make the pixel value of the pixel more opaque as the virtual projection line corresponding to the point is, and to make the pixel value of the pixel more transparent as the virtual projection line corresponds to the target point having a smaller number. In addition, as the number of imaging devices 2 is reduced as the determination condition, the position setting of the virtual camera may be limited. For example, when the number of imaging devices 2 is two as the determination condition, the installation position of the virtual camera may be limited so that it can be set only on a straight line connecting the installation positions of the two imaging devices 2.

  Reference Signs List 1 image monitoring system, 2 imaging device, 3 input device, 4 image generation device, 5 output device, 41 storage unit, 42 attention object image extraction unit, 43 attention object image restoration unit, 44 image output unit, 411 extraction object image, 412 virtual camera information, 413 determination area information, 414 real camera information, 431 projection line generation means, 432 object corresponding point detection means, 433 object image generation means.

Claims (4)

An object image of the three-dimensional object is extracted from each of the real images obtained by simultaneously imaging the three-dimensional object from different viewpoints by a plurality of real cameras, and a virtual camera image is generated by imaging the three-dimensional object with the virtual camera An image generation device that
A storage unit storing real camera information representing camera parameters of the real camera and virtual camera information representing camera parameters of the virtual camera;
Projection line generation means for obtaining a virtual projection line corresponding to each pixel of the virtual camera image using the virtual camera information;
An attention point is set on the virtual projection line, a projection position in each real image of the attention point is determined using the real camera information, and the projection position is within the object image in a plurality of real images. Object corresponding point detection means for detecting a point of interest as an object corresponding point;
Have a, and the object image generating means for generating an image of the three-dimensional object in the virtual camera image by the pixels corresponding to the virtual projection line the object corresponding point is detected,
The object corresponding point detection means
The surface position search for estimating the surface position of the three-dimensional object on the virtual projection line by setting the target points for each reference distance interval on the virtual projection line and searching for the target point to be the object corresponding point Processing and
The point of interest is sequentially set in the direction opposite to the virtual camera from the surface position on the virtual projection line, and the point of interest which is not the object corresponding point is searched for to set the solid object on the virtual projection line. Performing back surface position search processing to estimate the back surface position;
The object image generation means determines the pixel value of the pixel corresponding to the virtual projection line according to the object size on the virtual projection line and the object size in the viewing direction estimated from the back surface position.
A virtual camera image generation device characterized by   The virtual camera image generation according to claim 1, wherein the object corresponding point detection means sets the target points on each of the virtual projection lines at each reference distance interval smaller than the assumed size of the three-dimensional object. apparatus. The object corresponding point detecting means is smaller than the reference distance interval in a portion included in a predetermined near space around the object corresponding point detected in another virtual projection line among the virtual projection line The virtual camera image generation device according to claim 1 , wherein the attention point is set by an interval. The storage unit stores a three-dimensional space range in which the three-dimensional object can exist;
The virtual camera image generation apparatus according to any one of claims 1 to 3 , wherein the object corresponding point detection unit sets the target point only in the space range.

Images