WO2025225378A1 - Image processing device, image processing method, and program - Google Patents

Abstract

A first image processing device 20 sets the positions and orientations of a plurality of second virtual cameras on the basis of the optical axes of the plurality of first virtual cameras in a virtual space corresponding to the positions and orientations of a plurality of imaging devices in a real space, and generates a plurality of depth images indicating the distance between each of the plurality of second virtual cameras and the 3D model of the subject generated on the basis of the plurality of captured images acquired by the plurality of imaging devices.

Description

Image processing device, image processing method and program

The present disclosure relates to an image processing device for compressing and distributing 3D model data representing a 3D model.

There is a technology that generates 3D model data showing a 3D model of a subject (object) using multiple images captured by multiple imaging devices, and then generates a virtual viewpoint image as seen from a camera (virtual camera) virtually placed in a virtual space in which the 3D model exists. Recently, technology that generates 3D model data of a subject on a server, compresses and distributes the 3D model data, and allows users to operate a virtual camera on their own local terminal (client) such as a PC or tablet to display a virtual viewpoint image has been attracting attention.

In a technology for compressing and distributing 3D model data, the distributing device arranges multiple virtual cameras (or a group of virtual cameras) to surround the 3D model and generates depth images and texture images from each of the multiple virtual cameras. The generated depth and texture images are then compressed, and the compressed depth and texture images, along with information indicating the positions and orientations of the multiple virtual cameras, are distributed to the user. The receiving device decodes the compressed depth and texture images, and reconstructs the 3D model based on the decoded depth and texture images and information indicating the positions and orientations of the multiple virtual cameras.

Patent Document 1 describes a method for determining the viewpoints of a group of virtual cameras so as to minimize fluctuations in the position of key objects in depth images between frames, with the aim of improving the compression rate of depth images. By reducing the number of motion vectors included in the encoded stream when encoding depth images, it is expected that the compression rate of depth images will improve.

International Publication No. 2018/079260

However, depending on the position and orientation of the virtual cameras set on the device on the distribution side, there is a risk that the quality of the 3D model reconstructed on the device on the receiving side may be reduced. For example, if the shape of the 3D model generated on the device on the distribution side is complex, depending on the position and orientation of the virtual cameras, the complex shape of the 3D model may not be sufficiently represented in the depth image, and the shape accuracy of the 3D model reconstructed on the user side may be reduced. In other words, the shape of the 3D model generated on the distribution side may differ significantly from the shape of the 3D model reconstructed on the receiving side. Therefore, to reconstruct a 3D model more accurately, it is necessary to appropriately determine the position and orientation of the virtual cameras.

The present disclosure therefore aims to reduce the possibility of a decrease in the quality of the 3D model generated on the receiving device.

In order to solve the above problems, the image processing device according to the present disclosure has the following configuration: a setting means for setting the positions and orientations of multiple second virtual cameras based on the optical axes of multiple first virtual cameras in virtual space that correspond to the positions and orientations of multiple image capture devices in real space; and a generation means for generating multiple depth images that indicate the distance between each of the multiple second virtual cameras and a 3D model of a subject that is generated based on the multiple captured images acquired by the multiple image capture devices.

This disclosure reduces the possibility of a decrease in the quality of the 3D model generated on the receiving device.

1 is a block diagram showing an example of the configuration of an image processing system 1. FIG. FIG. 1 illustrates an example of a hardware configuration of an image processing apparatus. FIG. 2 is a diagram illustrating an example of the functional configuration of a first image processing device 20 and a second image processing device 30. FIG. 1 is a diagram illustrating an example of the configuration of an imaging system 10. FIG. 3 is a diagram showing a first virtual camera group 305 that corresponds to the physical camera group 302 and a 3D model 304 of a subject that is generated by the first virtual camera group 305. 3A and 3B are diagrams showing a second virtual camera group 307 set based on viewpoint information of a first virtual camera group 305. FIG. 10A and 10B are diagrams illustrating generation of viewpoint information of a virtual camera according to the first embodiment. 10 is a flowchart illustrating an example of a process for compressing and distributing 3D model data according to the first embodiment. 10 is a flowchart illustrating an example of a process for receiving 3D model data and generating a virtual viewpoint image according to the first embodiment. 10 is a flowchart illustrating an example of a process for generating viewpoint information of a group of virtual cameras according to the first embodiment. 10 is a flowchart illustrating an example of a restoration process of a 3D model according to the first embodiment. 10A and 10B are diagrams illustrating generation of viewpoint information of a group of virtual cameras according to the second embodiment. 10 is a flowchart illustrating an example of a process for compressing and distributing 3D model data according to a second embodiment. 10 is a flowchart illustrating an example of a restoration process of a 3D model according to a second embodiment.

According to a preferred embodiment of the present invention, the image processing device has a setting means for setting the positions and orientations of multiple second virtual cameras based on the optical axes of multiple first virtual cameras in virtual space that correspond to the positions and orientations of multiple imaging devices in real space. The image processing device also has a generation means for generating multiple depth images that indicate the distance between each of the multiple second virtual cameras and a 3D model of the subject generated based on multiple captured images acquired by the multiple imaging devices. Here, it is assumed that the multiple imaging devices are aligned with each other. The multiple first virtual cameras are set based on the positional relationships between the multiple imaging devices in real space. In other words, the multiple first virtual cameras in virtual space are reproductions in virtual space of the multiple imaging devices in real space.

In this manner, the positions and orientations of the multiple second virtual cameras are set based on the positions and orientations of the multiple imaging devices. Therefore, the subject depicted in the depth images of the multiple second virtual cameras is similar to the subject included in the captured images acquired from the multiple imaging devices. Specifically, the shape of the area representing the subject included in the depth images generated from the second virtual cameras is similar to the shape of the area representing the subject included in the captured images. The 3D model of the subject generated by the distribution device is generated based on the multiple captured images acquired from the multiple imaging devices. Therefore, if the depth images used to reconstruct (restore) the 3D model of the subject on the receiving device are similar to the captured images, the 3D model reconstructed on the receiving device will also be similar to the 3D model generated on the distribution device. In other words, it is possible to reduce the possibility that the shape of the 3D model generated on the receiving device will differ from that of the 3D model generated on the distribution device, i.e., the reproducibility of the 3D model will be degraded.

The image processing device also has an encoding means for encoding the depth images. The image processing device also has an output means for outputting the encoded depth images and viewpoint information indicating the positions and orientations of the second virtual cameras to another device that reconstructs the 3D model. Here, the other device is a device that reconstructs the 3D model based on the encoded depth images and viewpoint information.

Furthermore, in the above-mentioned image processing device, the generation means generates a virtual viewpoint image including a 3D model of the subject for each of the plurality of second virtual cameras. The encoding means then encodes the plurality of virtual viewpoint images, and the output means outputs the encoded plurality of virtual viewpoint images to the other device.

In this manner, the colors of each component of the subject included in the virtual viewpoint image (texture image) generated from the second virtual camera are close to the colors of each component of the subject included in the captured image. In other words, it is possible to reduce the possibility that the 3D model generated on the receiving device will have different colors than the 3D model generated on the transmitting device, i.e., the possibility that the reproducibility of the 3D model will be degraded.

Furthermore, the generation means generates correspondence information indicating whether each pixel in the texture image corresponds to a component of the 3D model of the subject, for each of the second virtual cameras. The encoding means then encodes the plurality of pieces of correspondence information, and the output means outputs the encoded plurality of pieces of correspondence information to the other device.

This aspect allows the receiving device to identify, among the pixels of the texture image, the pixels to be used to color the 3D model of the subject. For example, if a subject is occluded by another subject in an image captured by the imaging device, that captured image is used to generate a texture image for the second virtual camera, which could result in the generation of a texture image with color information that differs from the color information of that subject. Therefore, by generating correspondence information that indicates areas where occlusion is not thought to occur, the possibility of colors differing when the 3D model is reconstructed can be reduced.

Furthermore, the second virtual cameras are set at positions closer to the 3D model of the subject than the first virtual cameras.

Furthermore, the generation means generates one depth image for one second virtual camera. When distributing a 3D model showing movement over time, one depth image is generated for one second virtual camera at each of a plurality of time points.

According to another preferred embodiment of this embodiment, the image processing device has an acquisition means for acquiring a plurality of encoded depth images and first viewpoint information indicating the positions and orientations of a plurality of second virtual cameras. Here, a depth image is an image indicating the distance between each of the plurality of second virtual cameras and a 3D model of the subject generated based on a plurality of captured images acquired by the imaging device. The plurality of second virtual cameras are virtual cameras set based on the optical axes of a plurality of first virtual cameras in virtual space corresponding to the positions and orientations of the plurality of imaging devices in real space. The image processing device also has a decoding means for decoding the plurality of encoded depth images. The image processing device also has a generation means for generating a 3D model of the subject based on the plurality of decoded depth images and the first viewpoint information.

The image processing device also acquires second viewpoint information indicating the position and orientation of a third virtual camera different from the first virtual camera and the second virtual camera. The image processing device also generates a virtual viewpoint image based on the second viewpoint information and a 3D model of the subject. The third virtual camera may be set by a user operating an input device such as a joystick, or may be set based on the position of the generated 3D model.

This allows a virtual viewpoint image to be generated using a 3D model generated on the receiving device.

According to another preferred embodiment of this embodiment, a program causes a computer to execute the functions of the image processing device described above. By executing this program, the computer preferably functions as the image processing device described above.

<Example>
Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant descriptions will be omitted.

Note that a virtual viewpoint image is an image generated by a user freely manipulating the position and orientation of a virtual camera, and is also called a free viewpoint image or arbitrary viewpoint image. Furthermore, unless otherwise specified, the term "image" will be used to refer to both moving and still images.

Image processing system 1 is a system that generates a virtual viewpoint image that represents a scene from a specified virtual viewpoint, based on multiple images captured by multiple imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user; for example, the virtual viewpoint image also includes an image corresponding to a viewpoint selected by the user from multiple candidates. Furthermore, this embodiment will mainly describe a case where the virtual viewpoint is specified by user operation, but the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. Furthermore, this embodiment will mainly describe a case where the virtual viewpoint image is a video, but the virtual viewpoint image may also be a still image.

The viewpoint information used to generate a virtual viewpoint image is information that indicates the position and orientation (line of sight) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set that includes parameters that indicate the three-dimensional position of the virtual viewpoint and parameters that indicate the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set serving as viewpoint information may include a parameter that indicates the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets that respectively correspond to multiple frames that make up a video of the virtual viewpoint image, and may be information that indicates the position and orientation of the virtual viewpoint at each of multiple consecutive points in time.

The image processing system 1 has multiple imaging devices that capture images of an imaging area from multiple directions. The imaging area may be, for example, a stadium where sports such as soccer or karate are held, or a stage where concerts or plays are held. The multiple imaging devices are installed in different positions surrounding such an imaging area and capture images in sync. Note that the multiple imaging devices do not have to be installed around the entire perimeter of the imaging area; depending on installation location restrictions, they may be installed only around a portion of the perimeter of the imaging area. The number of imaging devices is not limited to the example shown in the figure; for example, if the imaging area is a soccer stadium, around 30 imaging devices may be installed around the stadium. Imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may also be installed.

Note that in this embodiment, the multiple imaging devices are each assumed to be cameras having an independent housing and capable of capturing images from a single viewpoint. However, this is not limited to this, and two or more imaging devices may be configured within the same housing. For example, a single camera equipped with multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple imaging devices.

A virtual viewpoint image can be generated, for example, using the following method. First, multiple images (multiple captured images) are obtained by capturing images from different directions using multiple imaging devices. Next, a foreground image is obtained from the multiple captured images, extracting a foreground area corresponding to a specific object, such as a person or a ball, and a background image is obtained from the multiple captured images, extracting a background area other than the foreground area. A foreground model representing the three-dimensional shape of the specific object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of the background, such as a stadium, is generated based on the background image. The texture data is then mapped to the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a virtual viewpoint image. However, the method for generating a virtual viewpoint image is not limited to this, and various methods can be used, such as generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model.

A foreground image is an image in which an object's area (foreground area) has been extracted from an image captured by an imaging device. An object extracted as a foreground area is a dynamic object (moving body) that moves (its absolute position and shape can change) when images are captured from the same direction in chronological order. Examples of objects include players, referees, and other people on the field where a sport is taking place, such as the ball in a ball game, or singers, musicians, performers, and presenters in a concert or entertainment event.

A background image is an image of at least an area (background area) that is different from the foreground object. Specifically, a background image is an image in which the foreground object has been removed from the captured image. Furthermore, the background refers to an object that remains stationary or nearly stationary when images are taken from the same direction in chronological order. Examples of such objects include a stage for a concert, a stadium where an event such as a sport is held, a structure such as a goal used in a ball game, or a field. However, the background is at least an area that is different from the foreground object, and the captured object may include other objects in addition to the object and background.

A virtual camera is a virtual camera that is different from the multiple imaging devices actually installed around the imaging area, and is a concept used to conveniently explain the virtual viewpoint involved in generating a virtual viewpoint image. In other words, a virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in this virtual image can be expressed as the position and orientation of the virtual camera. In other words, a virtual viewpoint image can be said to be an image that simulates the captured image obtained by a camera, assuming that the camera exists at the position of the virtual viewpoint set in space.

Example 1
In this embodiment, a process for determining the viewpoints of a group of virtual cameras for generating depth images and texture images to be distributed from a server to a client based on viewpoint information of a physical camera will be described. Note that in this embodiment, the receiving device will be referred to as the client. Also, in this embodiment, the imaging device arranged in real space will be referred to as the physical camera.

<Hardware configuration of image processing system>
FIG. 1A is a diagram showing an example of the overall configuration of an image processing system 1 according to this embodiment.

The image processing system 1 has a shooting system 10, a first image processing device 20, a second image processing device 30, an input device 40, and a display device 50. The image processing system 1 generates shape information of a 3D model of an object using images (plurality of captured images) taken by multiple physical cameras. The image processing system 1 then generates viewpoint information (first viewpoint information) of a second virtual camera group for generating depth images and texture images to be distributed from the server to the client. Furthermore, the image processing system 1 generates and compresses depth images and texture images based on the generated viewpoint information of the second virtual camera group, and distributes these images and the information necessary to reconstruct (restore) the 3D model to the user as 3D model data.

The photography system 10 places multiple physical cameras in different positions and synchronously photographs a subject (object) from multiple viewpoints. The multiple captured images obtained through synchronous photography, as well as viewpoint information (external/internal parameters, image size) for each physical camera of the photography system 10, are then transmitted to the first image processing device 20. The external parameters of a camera are parameters that indicate the position and orientation of the camera (e.g., rotation matrix and position vector). The internal parameters of a camera are internal parameters specific to the camera, such as focal length, image center, and lens distortion parameters. The external and internal parameters of a camera are collectively referred to as camera parameters. Image size refers to the width and height of the image. The viewpoint information of this group of physical cameras is used when determining the viewpoint information (camera parameters and image size) of the second group of virtual cameras used to generate depth images and texture images.

The first image processing device 20 generates a 3D model of the foreground object based on the multiple captured images input from the imaging system 10 and the viewpoint information for each physical camera. The foreground object (subject) is, for example, a person or moving object within the imaging range of the imaging system 10. The first image processing device 20 then generates viewpoint information for a second virtual camera group for generating depth images and texture images. Furthermore, the first image processing device 20 generates and compresses depth images and texture images based on the generated viewpoint information for the second virtual camera group, and outputs these compressed images and the information (metadata) required to restore the 3D model to the second image processing device 30. The metadata is, for example, the generated viewpoint information for the second virtual camera group.

First, the first image processing device 20 estimates shape information (shape information) of the subject based on multiple captured images and viewpoint information for each physical camera. Estimating the shape information of the subject uses, for example, a visual hull intersection method. Multiple virtual cameras (multiple first virtual cameras) are set in virtual space corresponding to the imaging system 10 existing in real space. These multiple first virtual cameras are referred to as a first virtual camera group. This first virtual camera group is a reproduction of a physical camera group in virtual space. In other words, the viewpoint information of the first virtual camera group corresponds to the viewpoint information of the physical camera group in real space. Then, shape information of the subject is generated by using the viewpoint information of the first virtual camera group and the multiple captured images in the visual hull intersection method. As a result of this processing, a 3D point cloud (a set of points with three-dimensional coordinates) that represents the shape information of the subject is obtained. This 3D point cloud is also called a visual hull. Note that the method of deriving shape information of the subject from captured images is not limited to this. Furthermore, the method of expressing the shape information of the subject is not limited to a 3D point cloud, but can also be a mesh or voxel. Furthermore, a texture image (color information) is determined for each point in the generated 3D point cloud of the subject using multiple captured images. Therefore, 3D model data representing a 3D model of the subject includes shape information indicating the shape and color information indicating the color. Note that in this embodiment, the 3D model is generated from shape information and color information, but this is not limited to this. For example, the 3D model includes shape information and may be managed as data separate from color information. Note that the method of setting a first virtual camera group in virtual space corresponding to a physical camera group in real space is well-known technology in the field of CG, etc., and therefore will not be described here.

Next, the first image processing device 20 generates viewpoint information (position and orientation) of multiple second virtual cameras for generating depth images and texture images to be distributed. Hereinafter, the multiple second virtual cameras will be referred to as a second virtual camera group. The position of this second virtual camera group is, for example, arranged on the optical axis of the first virtual camera group, and the orientation is set to be the same as the orientation of the first virtual camera. Since the viewpoint information of the first virtual camera group corresponds to the viewpoint information of the physical camera group in real space, the viewpoint information of the second virtual camera group is generated based on the viewpoint information of the physical camera group. The method for generating the viewpoint information of the second virtual camera group will be explained in detail later. Hereinafter, the viewpoint information of the second virtual camera group will be referred to as second viewpoint information. The first image processing device 20 generates a depth image of the 3D model based on the second viewpoint information of this second virtual camera group. Depth images are generated for the number of second virtual cameras. Specifically, each point in the 3D point cloud of the subject is projected onto the same plane as the imaging plane of the second virtual camera group. For each second virtual camera group, the distance (depth) from the second virtual camera to the subject is calculated for each projected pixel, and a depth value is set for each pixel of the depth image. Furthermore, the first image processing device 20 captures an image of the subject based on the second viewpoint information of the second virtual camera group and generates a texture image. The texture image is generated by blending the colors of multiple captured images while increasing the priority of pixel values of images captured by a physical camera with a line of sight close to the line of sight of the second virtual camera. This method of setting a high priority (weight) for pixel values of images captured by a physical camera with a line of sight close to the line of sight of the second virtual camera when blending multiple captured images is referred to as a virtual viewpoint-dependent texture image generation method. Details of the virtual viewpoint-dependent texture image generation method will be described later. This virtual viewpoint-dependent texture image is generated by selecting an image captured by a physical camera that determines the pixel value of the subject depending on the position and orientation of the virtual camera, so the color of the subject changes when the virtual camera moves. Therefore, a texture image generated using this method is referred to as a virtual viewpoint-dependent texture image. On the other hand, virtual-viewpoint-independent texture images are generated in a way that the pixel values of the subject do not change depending on the position and orientation of the virtual camera. In this embodiment, texture image generation deals with virtual-viewpoint-dependent texture images, but texture images may also be generated in a virtual-viewpoint-independent manner. An example of the process for generating virtual-viewpoint-dependent and virtual-viewpoint-independent texture images is shown below.

The process of generating a virtual viewpoint-dependent texture image includes, for example, a process of determining the visibility of points in the 3D point cloud that constitutes the subject, and a process of deriving colors based on the position and orientation of the virtual camera.

)。 In the visibility determination process, the physical camera that can capture each point is identified based on the positional relationship between each point in the 3D point cloud and the multiple physical cameras included in the group of physical cameras that the imaging system 10 has. In the color derivation process, for example, a point in the 3D point cloud is set as a focus point, and the color of that focus point is derived. Specifically, the following process is performed for each focus point. A focus point that is included in the imaging range of the second virtual camera is selected. Then, a first virtual camera that has a line of sight close to the line of sight of the second virtual camera whose imaging range includes that focus point and can capture that focus point is selected. Because the first virtual camera is a reproduction of a physical camera in virtual space, selecting the first virtual camera is the same as selecting a physical camera. The selected focus point is then projected onto the image captured by the selected physical camera. The color of the pixel at the projection destination is set as the color of that focus point. A physical camera is selected, for example, based on whether the angle between the line of sight from the second virtual camera to the focus point and the line of sight from the physical camera to the focus point is within a certain angle. If the point of interest can be captured by multiple physical cameras, multiple physical cameras with viewing directions close to the viewing direction of the second virtual camera are selected, and the point of interest is projected onto each of the images captured by those physical cameras. The pixel values of the projection destination are then obtained, and a weighted average is calculated so that pixel values from physical cameras with viewing directions close to the second virtual camera are used preferentially, thereby determining the color of the point of interest.

This process is performed while changing the focus point, and the color of the focus point is projected onto the same surface as the imaging surface of the second virtual camera, thereby generating a texture image that is dependent on the virtual viewpoint. While this has been explained in conjunction with the present embodiment, this is not limited to the above, and when generating a texture image that is dependent on a virtual camera different from the second virtual camera, the second virtual camera is changed to the different virtual camera and the above process is performed.

The process of generating a texture image independent of the virtual viewpoint includes, for example, the visibility determination process described above and a process of deriving a color that is independent of the position and orientation of the virtual camera. After the visibility determination process, for example, a point in the 3D point cloud is set as the focus point, and that focus point is projected onto an image captured by a physical camera corresponding to a first virtual camera that can capture images, and the color of the pixel at the projection destination is set as the color of that focus point.

If the point of interest can be captured by multiple first virtual cameras, the point of interest is projected onto each of the images captured by multiple physical cameras corresponding to the multiple first virtual cameras. The pixel values at the projection destination are then obtained and the average of the pixel values is calculated to determine the color of the point of interest. This process is performed while changing the point of interest, and the color of the point of interest is projected onto the same surface as the imaging surface of the second virtual camera, thereby generating a texture image that is independent of the virtual viewpoint.

In virtual viewpoint image generation technology, the method of generating color information dependent on a virtual viewpoint is known as a method of generating images with higher image quality than those independent of a virtual viewpoint, because priority is given to the pixel values of the image captured by a physical camera with a line of sight close to that of the second virtual camera. In addition, a virtual viewpoint-dependent virtual viewpoint image (texture image) generated by a virtual camera with a position and orientation similar to that of a physical camera is heavily influenced by the image captured by that physical camera, making it possible to generate a texture image with image quality close to that of the captured image. On the other hand, a virtual viewpoint-dependent texture image generated from the viewpoint of a virtual camera whose position and orientation is significantly different from that of the physical camera, or a texture image generated independent of a virtual viewpoint, is generated by interpolating pixel values using multiple physical cameras. As a result, the image may contain pixel values that differ from the color information of the actual subject, or may have low contrast.

Finally, the first image processing device 20 compresses (encodes) the depth image and texture image using a video compression method such as H.264 or H.265. Note that the compression method is not limited to video compression, and any method that can encode to a size smaller than the original data volume, such as file compression, may be used. The first image processing device 20 does not need to output depth images and texture images that do not show the subject to the second image processing device 30. The second image processing device 30 back-projects the depth image into virtual space based on the second viewpoint information of the second virtual camera, making it possible to reconstruct shape information of a 3D model. Furthermore, by using the pixel values of the texture image at the same coordinates as each depth value of the depth image as color information at the point where the depth value is back-projected, it is possible to add color information to the shape information and reconstruct a 3D model.

The first image processing device 20 may also extract a rectangular image of the subject's shooting range from the pre-compression depth image and texture image, compress this rectangular image (ROI image), and distribute it.

In this case, the metadata may include coordinate information for the cut-out rectangular image. The first image processing device 20 may also arrange the rectangular images to form a single image, compress it, and distribute it. By distributing the rectangular image instead of the entire image, it is possible to reduce the amount of data.

Furthermore, the first image processing device 20 may generate depth images containing high-precision depth values, such as single-precision floating-point numbers (32 bits), which cannot be used for video compression using H.264 or H.265. In this case, the video is compressed after converting the depth information to a precision that allows video compression (8 bits or 10 bits). The conversion method may involve, for example, performing scalar quantization processing, and compressing and distributing the quantized depth image. In this case, the minimum and maximum values of the depth value range before quantization may be included as metadata. By performing scalar quantization, the client can reconstruct a 3D model of the subject being photographed or the subject within the photographic range, which would not be possible with the precision required for video compression.

The second image processing device 30 receives and decodes the depth image, texture image, and metadata from the first image processing device 20 and restores the 3D model. As described above, the 3D model is restored by back-projecting the depth image and texture image into virtual space based on the second viewpoint information of the second virtual camera group contained in the metadata. The second image processing device 30 also calculates viewpoint information (second viewpoint information) of a third virtual camera for generating a virtual viewpoint image viewed by the user based on input values received from the input device 40, which will be described later. The second image processing device 30 then generates a virtual viewpoint image based on the calculated second viewpoint information and the restored 3D model. The second image processing device 30 then outputs the generated virtual viewpoint image to the display device 50.

The input device 40 accepts input values used by the user to set the third virtual camera and transmits the input values to the second image processing device 30. For example, the input device 40 has input units such as a joystick, jog dial, touch panel, keyboard, and mouse. The user setting the third virtual camera sets the position and orientation of the third virtual camera by operating the input unit. Note that in this embodiment, the user sets the position and orientation of the third virtual camera, but this is not limited to this, and the position and orientation of the third virtual camera may also be set using position information of the 3D model. The position information of the 3D model here may be generated by the first image processing device 20 on the distribution side, or may be generated by the second image processing device 30 on the receiving side.

The display device 50 displays the virtual viewpoint image generated and output by the second image processing device 30. The user views the virtual viewpoint image displayed on the display device 50 and sets the position and orientation of the virtual camera for the next frame via the input device 40.

FIG. 1B is a diagram showing an example of the hardware configuration of the first image processing device 20 of this embodiment. The hardware configuration of the second image processing device 30 is also similar to the configuration of the first image processing device 20 described below. The first image processing device 20 of this embodiment is composed of a CPU 101, RAM 102, ROM 103, and communication unit 104.

The CPU 101 controls the entire first image processing device 20 using computer programs and data stored in the RAM 102 and ROM 103, thereby realizing each function of the first image processing device 20 shown in Figures 1A and 1B. Note that the first image processing device 20 may have one or more pieces of dedicated hardware different from the CPU 101, and at least some of the processing by the CPU 101 may be performed by the dedicated hardware. Examples of dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The RAM 102 temporarily stores programs and data supplied from the auxiliary storage device 214, as well as data supplied from the outside via the communication unit 104. The ROM 103 stores programs that do not require modification.

The communication unit 104 is used for communication between the first image processing device 20 and an external device. For example, if the first image processing device 20 is connected to an external device via a wired connection, a communication cable is connected to the communication unit 104. If the first image processing device 20 has the function of communicating wirelessly with an external device, the communication unit 104 is equipped with an antenna.

<Functional configuration of image processing device>
FIG. 2 is a diagram showing an example of the functional configuration of the first image processing device 20 and the second image processing device 30. As shown in FIG.

The first image processing device 20 is composed of a shape information generation unit 201, a viewpoint determination unit 202, a depth image generation unit 203, a texture image generation unit 204, an encoding unit 205, and a distribution unit 206.

The shape information generation unit 201 uses the communication unit 104 to estimate shape information of the subject using the multiple captured images and viewpoint information of the physical cameras received from the imaging system 10. The shape information is estimated using the volume intersection method described above. Therefore, the shape information generation unit 201 also generates viewpoint information of a first virtual camera group in virtual space that corresponds to the physical camera group in real space from the acquired viewpoint information of the physical cameras. The shape information generation unit 201 outputs the estimated shape information to the viewpoint determination unit 202, depth image generation unit 203, and texture image generation unit 204. The shape information generation unit 201 also outputs the received viewpoint information of the physical cameras and multiple captured images to the viewpoint determination unit 202 and texture image generation unit 204.

Based on the input data from the shape information generation unit 201, the viewpoint determination unit 202 determines the viewpoints of the second virtual camera group and generates viewpoint information, with the aim of improving the quality of the 3D model restored by the second image processing device 30. The viewpoint determination unit 202 outputs the generated viewpoint information of the second virtual camera group to the depth image generation unit 203 and, via the depth image generation unit 203, to the texture image generation unit 204.

The viewpoint information of the second virtual camera group is generated, for example, by matching the position and orientation of the second virtual camera to that of the first virtual camera corresponding to the physical camera and dolly-zooming the second virtual camera in the line of sight of the first virtual camera. Specifically, the second virtual camera, placed at the position of the first virtual camera corresponding to the physical camera, approaches the subject in the line of sight of the first virtual camera while adjusting the focal length (zoom) so that the size of the subject in the virtual viewpoint image generated from the second virtual camera is maintained. This process of deriving the viewpoint information of the second virtual camera will be described in detail below using Figures 3A to 3C and 4. By performing this process while changing the physical camera, or while changing the subjects if there are multiple subjects in the captured image, it is possible to generate viewpoint information of the second virtual camera for each subject, for the number of physical cameras that captured the subject.

The viewpoint determination unit 202 may also change the image size (width and height) of the second virtual camera. First, the image size of the second virtual camera is set to the same as the image size of the physical camera, and after determining the position and orientation of the second virtual camera as described above, the image size of the second virtual camera is changed. For example, if the width and height of the virtual viewpoint image of the second virtual camera are changed to 1/N of the width and height of the image captured by the physical camera, the focal length and image center internal parameters of the second virtual camera are also multiplied by 1/N. In other words, if the image size of the physical camera is 4K (width 3840, height 2160) and the image size of the second virtual camera is changed to full HD (width 1920, height 1080), the width and height are each halved, the focal length and image center internal parameters of the second virtual camera are multiplied by 1/2. This allows the image size to be changed without changing the angle of view, reducing the amount of data sent from the server to the client.

The depth image generation unit 203 performs the aforementioned depth image generation process based on the shape information of the subject input from the shape information generation unit 201 and the viewpoint information of the second virtual camera group input from the viewpoint determination unit 202. The depth image generation unit 203 outputs the generated depth image and the viewpoint information of the second virtual camera group to the encoding unit 205.

The texture image generation unit 204 performs processing to generate the virtual viewpoint-dependent texture image described above based on input data from the shape information generation unit 201 and viewpoint information of the second virtual camera group input from the viewpoint determination unit 202 via the depth image generation unit 203. The texture image generation unit 204 outputs the generated texture image to the encoding unit 205. As described above, the virtual viewpoint-dependent texture image is generated by preferentially referencing the pixel values of the image captured by the physical camera that is closest in position and orientation to the second virtual camera. This makes it possible to generate high-quality texture images, and when the 3D model is restored on the client, a 3D model containing high-quality color information can be reproduced.

The texture image generation unit 204 may also generate a valid pixel map based on input data from the shape information generation unit 201 and viewpoint information of the second virtual camera group input from the viewpoint determination unit 202 via the depth image generation unit 203. The texture image generation unit 204 may output the generated valid pixel map to the encoding unit 205. The valid pixel map will be described in detail in Example 2.

The encoding unit 205 acquires the depth images and viewpoint information of the second virtual camera group input from the depth image generation unit 203, and the texture images input from the texture image generation unit 204. The encoding unit 205 compresses the depth images and texture images using the compression method described above, and outputs the compressed image group and viewpoint information (metadata) of the second virtual camera group to the distribution unit 206.

Furthermore, the encoding unit 205 may compress not only depth images and texture images, but also metadata. Compression methods include the file compression methods described above.

Furthermore, the encoding unit 205 may compress the effective pixel map input from the texture image generation unit 204 and output it to the distribution unit 206.

The distribution unit 206 uses the communication unit 104 to transmit the compressed depth image, texture image, and viewpoint information of the second virtual camera group input from the encoding unit 205 to the receiving unit 207, which will be described later.

The second image processing device 30 is composed of a receiving unit 207, a decoding unit 208, a 3D model restoration unit 209, a virtual camera control unit 210, and a virtual viewpoint image generation unit 211.

The receiving unit 207 uses the communication unit 104 to receive the compressed depth image, compressed texture image, and viewpoint information (metadata) of the second virtual camera group from the distribution unit 206, and outputs them to the decoding unit 208.

The decoding unit 208 decodes the compressed depth images and compressed texture images acquired from the receiving unit 207 and outputs them to the 3D model restoration unit 209 along with viewpoint information of the second virtual camera group. The decoding unit 208 may also decode metadata in addition to the depth images and texture images. Furthermore, the decoding unit 208 may decode an effective pixel map and output it to the 3D model restoration unit 209.

The 3D model restoration unit 209 restores a 3D model using the restoration method described above, based on the decoded depth image and decoded texture image acquired from the decoding unit 208, and the viewpoint information of the second virtual camera group. The restored 3D model is output to the virtual viewpoint image generation unit 211. Furthermore, the 3D model restoration unit 209 may generate color information for the 3D model using the effective pixel map acquired from the decoding unit 208. The method for using the effective pixel map will be explained in detail in Example 2.

The virtual camera control unit 210 uses the communication unit 104 to generate viewpoint information of a third virtual camera for generating a virtual viewpoint image from input values entered by the user via the input device 40, and outputs the viewpoint information of the third virtual camera to the virtual viewpoint image generation unit 211. The virtual camera control unit 210 may also output the generated viewpoint information of the user-specified third virtual camera to the 3D model restoration unit 209.

The virtual viewpoint image generation unit 211 generates a virtual viewpoint image based on the 3D model acquired from the 3D model restoration unit 209 and the viewpoint information of the third virtual camera acquired from the virtual camera control unit 210. The virtual viewpoint image is generated by placing a 3D model of the subject, a 3D model of the background object, and the third virtual camera in a virtual space, and generating an image seen from the third virtual camera. The 3D model of the background object is, for example, a CG (Computer Graphics) model created separately to be combined with the subject, and is created in advance and stored in the second image processing device 30 (for example, stored in ROM 103 in Figure 1B).

The 3D model of the subject and the 3D model of the background object are rendered using an existing CG rendering method. The virtual viewpoint image generation unit 211 transmits the generated virtual viewpoint image to the display device 50.

<Description of an example of generating viewpoint information of a group of second virtual cameras based on viewpoint information of a group of physical cameras>
Here, a method for generating viewpoint information of a second virtual camera group using viewpoint information of a physical camera group will be described with reference to Figures 3A to 3C and 4. Figures 3A to 3C are schematic diagrams for explaining an example of a method for generating viewpoint information of a second virtual camera group.

FIG. 3A shows a group of physical cameras 302 and a subject 301 arranged in real space. The subject 301 is photographed by the group of physical cameras 302. Note that a straight line 303 indicates the optical axis of each of the group of physical cameras 302.

FIG. 3B shows a diagram in which a first virtual camera group 305 corresponding to the physical camera group 302 is set and a 3D model 304 of the subject is generated by the first virtual camera group 305. The first image processing device 20 generates the 3D model 304 of the subject using multiple captured images acquired from the physical camera group 302. In order to generate a 3D model 304 in virtual space for the subject 301 in real space, a first virtual camera group 305 is generated in virtual space corresponding to the physical camera group 302 in real space. In other words, the first virtual camera group 305 is a virtual space reproduction of the physical camera group 302 in real space. Therefore, the optical axis 306 of the first virtual camera group 305 corresponds to the optical axis 303 of the physical camera group 302. The first image processing device 20 then generates the 3D model 304 of the subject 301 using viewpoint information from the generated first virtual camera group 305.

3C is a diagram showing the second virtual camera group 307 set based on the viewpoint information of the first virtual camera group 305. The second virtual camera group 307 is set on the optical axis 306 of the first virtual camera group 305. However, this is not limited to this, and the second virtual camera 307 may be set at a position closer to the optical axis 306. The second virtual camera group 307 is also set at a position closer to the 3D model 304 than the first virtual camera group 305. Generally, the second virtual camera group 307 used to generate depth images and texture images for distribution places a bounding box that encompasses the 3D model 304, and places the second virtual camera group 307 on a spherical surface surrounding the bounding box so that it faces the subject. This determines the position and orientation (extrinsic parameters) of the second virtual camera group 307. Then, the focal length (internal parameters) at which the entire 3D model 304 can be captured by the second virtual camera group 307 is determined. These are set manually, and the number and spacing of the second virtual cameras are often determined heuristically. If the 3D model 304, i.e., the subject 301, has a complex shape, occlusion occurs, whereby a portion of the 3D model 304 is hidden by another portion as viewed from a certain second virtual camera. Therefore, the number and placement of the second virtual cameras 307 must be appropriately determined. For a subject with a complex shape, if the number of second virtual cameras is small and their placement is inappropriate, resulting in frequent occlusion, the 3D model restored by the second image processing device 30 may differ significantly in shape from the 3D model 304 before distribution. However, increasing the number of second virtual cameras in an attempt to accurately restore the 3D model 304 increases the number of depth images and texture images to be distributed. Therefore, if a user attempts to restore the 3D model in an environment with low transmission bandwidth or on a local terminal with low processing performance, the frame rate may decrease. It is desirable for the 3D model displayed on the display device 50 to have higher image quality and a smooth frame rate, such as 60 fps, to provide the user with a high sense of realism. Therefore, it is necessary to appropriately set the number and arrangement of the second virtual camera group 307 and deliver depth images and high-quality texture images that can accurately reconstruct the 3D model 304 with a small amount of data. Therefore, the first image processing device 20 generates viewpoint information for the second virtual camera group 307 based on the viewpoint information of the physical camera group 302 described above. That is, the second virtual camera group 307 is arranged at the position of the first virtual camera group 305 corresponding to the physical camera group 302. The position of the second virtual camera group 307 is then set by dolly-zooming the second virtual camera group 307 onto the optical axis 306 of the first virtual camera 305 while maintaining the size of the subject on the screen so as to approach the 3D model 304 of the subject 301. It is assumed that the position to which the second virtual camera group 307 is dolly-zoomed is predetermined. For example, dolly-zooming may be performed until the distance from the 3D model 304 reaches a predetermined value. Alternatively, dolly-zooming may be performed until the size of the 3D model 304 projected on the imaging surface of the second virtual camera group 307 reaches a predetermined size. Note that the method for determining the viewpoint information of the second virtual camera group 307 is not limited to this method. Alternatively, the second virtual camera group 307 may simply be positioned near the optical axis 306 of the first virtual camera group 305, and internal parameters such as focal length and image size may be determined manually. That is, the viewpoint information of the second virtual camera group 307 may be determined based on the viewpoint information of the physical camera group 302. In virtual viewpoint generation technology, the physical camera group 302 is positioned in the number required to estimate the shape of the subject 301, surrounding the subject 301 using the volume intersection method described above, for example. Therefore, by sending depth images as observed by the physical camera group 302, the user can accurately restore the shape information of the shape-estimated 3D model 304. Furthermore, by delivering virtual viewpoint-dependent texture images with image quality similar to that of the physical camera group 302, the user can restore high-quality color information, allowing the user to play back a high-quality 3D model.

The method for determining viewpoint information for each virtual camera will be explained using Figure 4. Figure 4 is a schematic diagram in which a first virtual camera 401 captures a 3D model 402 of a subject and generates viewpoint information for a second virtual camera 403 based on the viewpoint information of the first virtual camera 401. The second virtual camera 403 moves along the optical axis of the first virtual camera 401 while dolly zooming to approach the 3D model 402. The second virtual camera 403 is then positioned at a position where it can capture the 3D model 402 within an area 404 that encompasses the 3D model 402 and allows the distance information (depth information) between the 3D model and the second virtual camera 403 to be expressed in 10 bits. Note that the area 404 is not limited to a cube, and may be a sphere centered on the 3D model 402. The first virtual camera 401 captures the 3D model 402 and generates a captured image 405. The first image processing device 20 estimates the 3D model 402 based on the captured images 405 captured by the physical cameras of the imaging system 10. The estimated 3D model 402 is then photographed by a second virtual camera 403 to generate a depth image 406 and a texture image 407. The size of the 3D model 402 depicted in the depth image 406 and the texture image 407 is the same as the size of the 3D model 402 depicted in the photographed image 405. As described above, the determination of the viewpoint information of the second virtual camera 403 is not limited to dolly zoom, and the size of the 3D model 402 depicted in the depth image 406 may be different from the size of the 3D model 402 depicted in the photographed image 405. Furthermore, although the second virtual camera 403 has been described as moving on the optical axis of the first virtual camera 401, the position and orientation of the second virtual camera 403 need only be close to the position and orientation of the first virtual camera 401, and it does not necessarily have to move on the optical axis of the first virtual camera 401. Furthermore, the determination of the viewpoint information of the second virtual camera 403 may be performed once when the second image processing device 30 is initialized, or may be performed for each frame in accordance with the movement of the 3D model 402.

<Compression and distribution of 3D model data and control of generation of virtual viewpoint images>
Fig. 5 is a flowchart showing the flow of processing for controlling the compression and distribution of 3D model data in the first image processing device 20 according to this embodiment. The flow shown in Fig. 5 is realized by reading a control program stored in the ROM 103 into the RAM 102 and executing it by the CPU 101. Execution of the flow in Fig. 5 is triggered when the shape information generation unit 201 receives a plurality of captured images and viewpoint information of the physical cameras from the imaging system 10.

In S501, the shape information generation unit 201 estimates and generates shape information of the subject based on multiple captured images. The generated shape information, viewpoint information of the physical camera group, and multiple captured images are output to the viewpoint determination unit 202 and texture image generation unit 204. The generated shape information is also output to the depth image generation unit 203. Note that the viewpoint information of the first virtual camera group corresponding to the physical camera group used to generate the subject is assumed to have been generated in advance. For example, when the physical camera group is placed as preparation before filming begins, a first virtual camera group corresponding to the placed physical camera group is generated. Shape information of the subject is generated using the viewpoint information of this first virtual camera group and multiple captured images.

In S502, the viewpoint determination unit 202 generates viewpoint information for the second virtual camera group based on the viewpoint information for the physical camera group. The generated viewpoint information for the second virtual camera group is output to the depth image generation unit 203 and to the texture image generation unit 204 via the depth image generation unit 203.

The process of generating the viewpoint information of the second virtual camera group is explained in Figure 7. Note that the generated viewpoint information of the second virtual camera group may be output to the texture image generation unit 204 without going through the depth image generation unit 203.

In S503, the depth image generation unit 203 generates a depth image of the 3D model based on the data acquired from the shape information generation unit 201 and the viewpoint determination unit 202. The generated depth image is output to the encoding unit 205.

In S504, the texture image generation unit 204 generates a texture image of the 3D model based on the data acquired from the shape information generation unit 201 and the viewpoint determination unit 202. The generated texture image is output to the encoding unit 205.

In S505, the encoding unit 205 encodes the depth image and texture image acquired from the depth image generation unit 203 and texture image generation unit 204. The encoded data is output to the distribution unit 206.

In S506, the distribution unit 206 distributes 3D model data including the data acquired from the depth image generation unit 203 and the encoding unit 205 and the viewpoint information of the second virtual camera group, and this flow ends. In the image processing system 1, the 3D model data is distributed to the second image processing device 30, but this is not limited to this. The 3D model data may also be distributed to a separate server that stores it.

FIG. 6 is a flowchart showing the flow of processing for generating a virtual viewpoint image using a depth image and a texture image in the second image processing device 30 according to this embodiment. Execution of the flow in FIG. 6 is triggered when the virtual camera control unit 210 receives an input value from the input device 40.

In S601, the virtual camera control unit 210 generates viewpoint information of a third virtual camera designated by the user based on input values from the input device 40. The generated viewpoint information of the third virtual camera is output to the virtual viewpoint image generation unit 211.

In S602, the receiving unit 207 receives the 3D model data distributed from the distribution unit 206. The received 3D model data is output to the decoding unit 208.

In S603, the decoding unit 208 decodes the depth images and texture images included in the 3D model data acquired from the receiving unit 207. Furthermore, if the viewpoint information of the second virtual camera group has also been encoded, the viewpoint information of the second virtual camera group is also decoded. The decoded depth images and texture images, as well as the viewpoint information of the second virtual camera group, are then output to the 3D model restoration unit 209.

In S604, the 3D model restoration unit 209 restores a 3D model of the subject based on the decoded depth image and texture image acquired from the decoding unit 208 and the viewpoint information of the second virtual camera group. The restored 3D model is output to the virtual viewpoint image generation unit 211. The 3D model restoration process is explained in Figure 8.

In S605, the virtual viewpoint image generation unit 211 generates a virtual viewpoint image based on the viewpoint information of the third virtual camera acquired from the virtual camera control unit 210 and the 3D model acquired from the 3D model restoration unit 209, and this flow ends.

After this flow is completed, the generated virtual viewpoint image is sent to the display device 50 and displayed on the display device 50.

The above describes the control of the compression and distribution of 3D model data and the generation of virtual viewpoint images in this embodiment.

<Description of the Process for Generating Viewpoint Information of Second Virtual Camera Group>
7 is an example of a flowchart showing the flow of processing for generating viewpoint information of the second virtual camera group according to this embodiment. Here, the method for generating viewpoint information of the second virtual camera group will be described as a method for generating viewpoint information based on the dolly zoom method of the second virtual camera described with reference to FIG. 4 . That is, the second virtual camera moves on the optical axis of the first virtual camera from the position of the first virtual camera corresponding to the position of the physical camera in a direction approaching the 3D model of the subject. Furthermore, a method will be described in which the second virtual camera adjusts the focal length while maintaining the size of the subject on the imaging plane of the second virtual camera.

The flow shown in FIG. 7 is executed by the viewpoint determination unit 202. Execution of FIG. 7 is triggered by the reception of a 3D model of the subject, viewpoint information of the physical camera group, and multiple captured images from the shape information generation unit 201. The flow in FIG. 7 provides a detailed explanation of the control for determining the viewpoint of the second virtual camera group based on the viewpoint information of the physical camera group in S502 of FIG. 5. When executing the flow shown in FIG. 7, it is assumed that viewpoint information of the first virtual camera group and a 3D model of the subject have been generated by the first image processing device 20.

In S701, a 3D model of the subject, viewpoint information of the first virtual camera group, and multiple captured images are acquired. Note that viewpoint information of the physical camera group may be acquired without acquiring viewpoint information of the first virtual camera group. In that case, viewpoint information of the first virtual camera group is generated using the viewpoint information of the physical camera group.

In S702, S703 and S704 are repeated for each physical camera.

In S703, S704 is repeated for each subject included in the multiple captured images. Subject recognition is performed based on the results of a face detection algorithm, person detection algorithm, etc. Alternatively, the area of each subject in the captured images can be identified by projecting a 3D model generated separately for each subject onto the same plane as the imaging surface of the physical camera using the viewpoint information of the physical camera.

In S704, viewpoint information for the second virtual camera is generated based on the 3D model and viewpoint information for the first virtual camera. The viewpoint information for the second virtual camera is generated using the method described with reference to Figure 4. The above process is repeated as described in S702 and S703 to generate viewpoint information for the second virtual camera group.

In S705, the viewpoint information of the generated second virtual camera group is sent to the depth image generation unit 203 and the texture image generation unit 204.

<Explanation of 3D model restoration control>
FIG. 8 is an example of a flowchart showing the flow of a 3D model restoration process according to this embodiment. Here, the 3D model restoration process is described based on a method for generating a 3D model including virtual viewpoint-dependent color information. In other words, the pixel values of the texture image of the second virtual camera group, which are closest to the position and orientation of the user-specified third virtual camera, are given priority as the color information for the 3D model. The flow shown in FIG. 8 is executed by the 3D model restoration unit 209. The flow in FIG. 8 provides a detailed description of the control for restoring a 3D model of a subject based on the decoded data in FIG. 6.

In S801, the 3D model restoration unit 209 acquires a depth image, a texture image, and viewpoint information of the second virtual camera group.

In S802, the 3D model restoration unit 209 projects the depth values of each pixel in the depth image into virtual space based on the external and internal parameters of the virtual camera corresponding to the depth image, and generates components of the shape information of the subject. For example, if the 3D model of the subject is represented by a 3D point cloud, each point becomes a component. By performing the above process on all of the multiple acquired depth images, the shape information of the subject is restored.

In S803, if the components of the shape information projected into the virtual space in the depth image are captured using multiple texture images, the pixel values of the texture image of the virtual camera with the closest position and orientation to the user-specified virtual camera are prioritized and determined as the color information for the components. This process is repeated for all components to restore the color information corresponding to the shape information. For example, if the shape information is a point cloud and the components are each point in the point cloud, the color information corresponding to all points is restored.

In S804, the restored 3D model is output to the virtual viewpoint image generation unit 211.

As explained above, in this embodiment, viewpoint information for the second virtual camera group is generated based on viewpoint information from the physical camera group, and a depth image and a texture image are generated using the generated viewpoint information for the second virtual camera group. This type of processing makes it possible to restore a high-quality 3D model on the receiving side without increasing the amount of data, even for 3D models with complex shapes. Note that while this embodiment describes the delivery of compressed 3D model data, the compressed 3D model data may be stored, or may be delivered in response to a client request. Also, while the example described uses a method in which a 3D model containing virtual-viewpoint-dependent color information is generated when the 3D model is restored, this is not limiting. After the 3D model's shape information is restored, color information for the 3D model may be generated simultaneously with the generation of a virtual viewpoint image. In this case, it is not necessary to assign color information to all of the 3D model's shape information; color information only needs to be generated for the shooting range (viewing angle) of the user-specified third virtual camera.

Example 2
In Example 1, a process was described in which viewpoint information of a second virtual camera group is generated based on viewpoint information of a physical camera group, and a depth image and a texture image are generated using the generated viewpoint information of the second virtual camera group. Next, Example 2 will be described, which illustrates an aspect in which an effective pixel map is generated in addition to a depth image and a texture image to deal with cases in which a subject is occluded by another subject or object. Note that the description of parts common to Example 1, such as the hardware configuration and functional configuration of the image processing device, will be omitted or simplified.

FIG. 9 is a schematic diagram showing an example of a method for generating an effective pixel map according to this embodiment. A subject is photographed using a physical camera, and a captured image 903 is generated. At this time, the subject in the captured image 903 has part of its body hidden by an obstruction. The obstruction is considered to be the subject.

The first image processing device 20 estimates the shapes of the subject and obstructing object using multiple captured images taken by the physical cameras of the imaging system 10, and generates shape information for each. That is, it generates a 3D model 901 of the subject and a 3D model 904 of the obstructing object. Next, the first image processing device 20 generates viewpoint information for the second virtual camera group described in Example 1 based on data such as viewpoint information and shape information from the physical camera group. The second virtual camera 905 is positioned on the optical axis of the first virtual camera 902, within an area 906 that encompasses the 3D model 901 of the subject and is within a range in which the depth of the subject can be expressed in 10 bits. The focal length of the second virtual camera 905 is adjusted so that the size of the subject 901 as viewed from the first virtual camera 902 is maintained. After generating the viewpoint information for the second virtual camera 905, the first image processing device 20 generates a depth image 907 and a texture image 908. Here, for each pixel value of the subject 901 appearing in the generated texture image 908, the pixel values of the image captured by the first virtual camera 902 are used preferentially in the area not obscured by the 3D model 904 of the obstructing object (the left side of the subject). On the other hand, in the area obscured by the 3D model 904 of the obstructing object (the right side of the subject), the pixel values of the image captured by the first virtual camera, which has a different viewpoint from the first virtual camera 902, are used. Therefore, there is a possibility that the image quality of the right and left sides of the subject appearing in the texture image 908 will differ significantly. If the color information of the 3D model is restored using this texture image 908, the image quality of parts of the 3D model will be reduced, and this part will stand out, which may cause the user to feel uncomfortable. Therefore, an effective pixel map 909 is generated that indicates high-quality areas of the texture image. For example, the effective pixel map assigns pixel values of 1 to unobstructed areas and 0 to obstructed areas. The image size of the effective pixel map 909 is the same as the image size of the texture image 908. The determination of whether or not an area is an occluded area is made, for example, based on whether each point constituting the shape information of the 3D model 901 of the subject is visible to the first virtual camera 902 using the visibility determination process described above. In other words, due to the visibility determination process, pixel values of the area of the occluding object 904 captured in the captured image 903 are not used as color information of the 3D model 901 of the subject. Therefore, a region of a texture image determined using pixel values of the captured image of the physical camera from which the viewpoint information of the second virtual camera 905 is generated is identified, and in the effective pixel map, pixel values of the identified area are set to 1, and other areas are set to 0. Note that the method of determining an occluded area is not limited to this. When restoring color information of the 3D model, the second image processing device 30 prioritizes pixel values of areas of the texture image corresponding to areas with a value of 1 in the effective pixel map and uses them as color information of the 3D model. Specifically, when generating color information for the restored shape information in a virtual viewpoint-dependent manner, the second image processing device 30 uses pixel values of the texture image corresponding to 1 in the effective pixel map and does not use pixel values of the texture image corresponding to 0 in the effective pixel map. However, if the shape information to which color information is to be added is stored only in the texture image corresponding to 0 in the effective pixel map, the pixel value of that texture image is used. While the effective pixel map has been described so far as having binary pixel values of 0 or 1, it can also be multi-valued. In this case, the pixel values of the effective pixel map can be used to weight the color information generated for the 3D model. In other words, the priority of the pixel values of the texture image is determined according to the pixel values of the effective pixel map. When generating a multi-valued effective pixel map with 255 levels, for example, the pixel values of the subject's outline or the boundary with an obstructing object can be set to 0, and can linearly increase to 255 as the subject approaches a certain distance (e.g., 5 px) inward toward the subject. This reduces the influence of unreliable pixel values of the texture image at the outline or boundary with an obstructing object when restoring the color information of the 3D model, allowing for the generation of color information for the 3D model using highly reliable pixel values.

FIG. 10 is a flowchart showing the flow of processing for controlling the compression and distribution of 3D model data in the first image processing device 20 according to this embodiment. Execution of the flow in FIG. 10 begins when the shape information generation unit 201 receives multiple captured images and viewpoint information from the physical cameras from the imaging system 10.

S1001 to S1003 are the same as S501 to S503 in Figure 5.

In S1004, the texture image generation unit 204 generates a texture image and a valid pixel map of the foreground model based on the data acquired from the shape information generation unit 201 and the viewpoint determination unit 202. The generated texture image and valid pixel map are output to the encoding unit 205.

In S1005, the encoding unit 205 encodes the depth image, texture image, and valid pixel map acquired from the depth image generation unit 203 and texture image generation unit 204. The encoded depth image, texture image, and valid pixel map are output to the distribution unit 206.

In S1006, the distribution unit 206 transmits 3D model data including the depth image, texture image, effective pixel map, and viewpoint information of the second virtual camera group acquired from the depth image generation unit 203 and encoding unit 205 to the reception unit 207, and this flow ends.

FIG. 11 is an example of a flowchart showing the flow of the 3D model restoration process according to this embodiment. The flow in FIG. 11 is executed by the 3D model restoration unit 209. The flow in FIG. 11 provides a detailed explanation of the control for restoring a 3D model of a subject based on the data decoded in S604 in FIG. 6.

In S1101, a depth image, texture image, effective pixel map, and viewpoint information of the second virtual camera group are obtained.

In S1102, the depth value of each pixel in the depth image is projected into virtual space based on the external parameters and internal parameters of the second virtual camera corresponding to the depth image, and shape information of the subject is restored.

In S1103, if the shape information is captured from multiple texture images, the pixel values of the texture image of the second virtual camera, which is closest in position and orientation to the user-specified third virtual camera, are given priority as color information. At this time, the priority of the pixel values of the texture image is determined according to the pixel values of the effective pixel map that correspond to the pixel values of the texture image. If the shape information is represented by a 3D point cloud, this process is repeated for all points in the point cloud to generate color information corresponding to the shape information.

S1104 is the same as S804 in Figure 8.

After this flow is completed, the virtual viewpoint image generation unit 211 generates a virtual viewpoint image based on the restored 3D model and viewpoint information of the third virtual camera specified by the user, and the generated virtual viewpoint image is displayed on the display device 50.

The above explains the compression and distribution of 3D model data including valid pixel maps, and the control of 3D model restoration. This type of processing makes it possible to restore a 3D model using highly reliable pixel values from texture images, even when the subject is hidden by an obstruction.

<Other Examples>
The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

The disclosure of this embodiment includes the following configurations, methods, systems, and programs.

(Configuration 1)
a setting means for setting positions and orientations of a plurality of second virtual cameras based on optical axes of a plurality of first virtual cameras in a virtual space corresponding to positions and orientations of a plurality of imaging devices in a real space;
and a generation means for generating a plurality of depth images indicating the distance between each of the plurality of second virtual cameras and a 3D model of a subject generated based on a plurality of captured images acquired by the plurality of imaging devices.

(Configuration 2)
2. The image processing device according to configuration 1, wherein the plurality of second virtual cameras are set on the optical axes of the plurality of first virtual cameras.

(Configuration 3)
2. The image processing device according to claim 1, further comprising encoding means for encoding the depth image.

(Configuration 4)
The image processing device according to configuration 3 further comprises an output means for outputting the encoded depth images and viewpoint information indicating the positions and orientations of the second virtual cameras to another device that reconstructs the 3D model based on the encoded depth images and the viewpoint information.

(Configuration 5)
the generating means generates a virtual viewpoint image including a 3D model of the subject for each of the second virtual cameras;
the encoding means encodes the plurality of virtual viewpoint images;
5. The image processing device according to configuration 4, wherein the output means outputs the encoded virtual viewpoint images to the other device.

(Configuration 6)
the generating means generates correspondence information indicating whether each pixel in the virtual viewpoint image corresponds to a component of a 3D model of the subject, for each of the plurality of second virtual cameras;
the encoding means encodes a plurality of pieces of correspondence information;
6. The image processing apparatus according to configuration 5, wherein the output means outputs the encoded correspondence information to the other apparatus.

(Configuration 7)
2. The image processing device according to configuration 1, wherein the second virtual cameras are set at positions closer to the 3D model of the subject than the first virtual cameras.

(Configuration 8)
2. The image processing device according to claim 1, wherein the generating means generates one depth image for one second virtual camera.

(Configuration 9)
an acquisition means for acquiring a plurality of encoded depth images indicating the distance between each of a plurality of second virtual cameras set based on the optical axes of a plurality of first virtual cameras in a virtual space corresponding to the positions and orientations of a plurality of image capturing devices in real space and a 3D model of a subject generated based on a plurality of captured images acquired by the image capturing devices, and first viewpoint information indicating the positions and orientations of the plurality of second virtual cameras;
a decoding means for decoding the encoded depth images;
a generating means for generating a 3D model of the subject based on the decoded depth images and the first viewpoint information;
1. An image processing device comprising:

(Configuration 10)
acquiring second viewpoint information indicating a position and an attitude of a third virtual camera different from the first virtual camera and the second virtual camera;
10. The image processing device according to configuration 9, wherein the generating means generates a virtual viewpoint image based on the second viewpoint information and a 3D model of the subject.

(Method 1)
a setting step of setting positions and orientations of a plurality of second virtual cameras based on optical axes of a plurality of first virtual cameras in a virtual space corresponding to positions and orientations of a plurality of imaging devices in a real space;
and a generation step of generating a plurality of depth images indicating the distance between each of the plurality of second virtual cameras and a 3D model of the subject generated based on a plurality of captured images acquired by the plurality of imaging devices.

(Method 2)
an acquisition process for acquiring a plurality of encoded depth images indicating the distance between each of a plurality of second virtual cameras set based on the optical axes of a plurality of first virtual cameras in a virtual space corresponding to the positions and orientations of a plurality of imaging devices in real space and a 3D model of a subject generated based on a plurality of captured images acquired by the imaging devices, and first viewpoint information indicating the positions and orientations of the plurality of second virtual cameras;
a decoding step of decoding the encoded depth images;
a generation step of generating a 3D model of the subject based on the decoded depth images and the first viewpoint information;
An image processing method comprising:

(program)
11. A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 10.

The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to clarify the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2024-073062, filed April 26, 2024, the entire contents of which are incorporated herein by reference.

202 viewpoint determination unit 203 depth image generation unit 204 texture image generation unit 205 encoding unit

Claims (13)

a setting means for setting positions and orientations of a plurality of second virtual cameras based on optical axes of a plurality of first virtual cameras in a virtual space corresponding to positions and orientations of a plurality of imaging devices in a real space;
and a generation means for generating a plurality of depth images indicating the distance between each of the plurality of second virtual cameras and a 3D model of a subject generated based on a plurality of captured images acquired by the plurality of imaging devices. The image processing device described in claim 1, characterized in that the multiple second virtual cameras are set on the optical axes of the multiple first virtual cameras. The image processing device according to claim 1, further comprising an encoding means for encoding the depth image. The image processing device described in claim 3, further comprising an output means for outputting the encoded depth images and viewpoint information indicating the positions and orientations of the second virtual cameras to another device that reconstructs the 3D model based on the encoded depth images and the viewpoint information. the generating means generates a virtual viewpoint image including a 3D model of the subject for each of the second virtual cameras;
the encoding means encodes the plurality of virtual viewpoint images;
5. The image processing apparatus according to claim 4, wherein the output means outputs the encoded virtual viewpoint images to the other device. the generating means generates correspondence information indicating whether each pixel in the virtual viewpoint image corresponds to a component of a 3D model of the subject, for each of the plurality of second virtual cameras;
the encoding means encodes a plurality of pieces of correspondence information;
6. The image processing apparatus according to claim 5, wherein said output means outputs the encoded correspondence information to said other apparatus. The image processing device described in claim 1, characterized in that the multiple second virtual cameras are set at positions closer to the 3D model of the subject than the multiple first virtual cameras. The image processing device described in claim 1, characterized in that the generation means generates one depth image for one second virtual camera. an acquisition means for acquiring a plurality of encoded depth images indicating the distance between each of a plurality of second virtual cameras set based on the optical axes of a plurality of first virtual cameras in a virtual space corresponding to the positions and orientations of a plurality of image capturing devices in real space and a 3D model of a subject generated based on a plurality of captured images acquired by the image capturing devices, and first viewpoint information indicating the positions and orientations of the plurality of second virtual cameras;
a decoding means for decoding the encoded depth images;
a generating means for generating a 3D model of the subject based on the decoded depth images and the first viewpoint information;
1. An image processing device comprising: acquiring second viewpoint information indicating a position and an attitude of a third virtual camera different from the first virtual camera and the second virtual camera;
The image processing apparatus according to claim 9 , wherein the generating means generates a virtual viewpoint image based on the second viewpoint information and a 3D model of the subject. a setting step of setting positions and orientations of a plurality of second virtual cameras based on optical axes of a plurality of first virtual cameras in a virtual space corresponding to positions and orientations of a plurality of imaging devices in a real space;
and a generation step of generating a plurality of depth images indicating the distance between each of the plurality of second virtual cameras and a 3D model of the subject generated based on a plurality of captured images acquired by the plurality of imaging devices. an acquisition process for acquiring a plurality of encoded depth images indicating the distance between each of a plurality of second virtual cameras set based on the optical axes of a plurality of first virtual cameras in a virtual space corresponding to the positions and orientations of a plurality of imaging devices in real space and a 3D model of a subject generated based on a plurality of captured images acquired by the imaging devices, and first viewpoint information indicating the positions and orientations of the plurality of second virtual cameras;
a decoding step of decoding the encoded depth images;
a generation step of generating a 3D model of the subject based on the decoded depth images and the first viewpoint information;
An image processing method comprising: A program for causing a computer to function as each of the means of the image processing device described in any one of claims 1 to 10.