TWI812548B - Method and computer device for generating a side-by-side 3d image - Google Patents

Abstract

A method for generating a side-by-side three-dimensional (3D) image based on a raw image is provided. The method includes the following steps: creating a 3D mesh; estimating depth information of the raw image; updating the left mesh area and the right mesh area of the 3D mesh based on the estimated depth information of the raw image; projecting each of the mesh vertices of the left mesh area onto a coordinate system of the side-by-side 3D image based on a left eye position, and projecting each of the mesh vertices of the right mesh area onto the coordinate system of the side-by-side 3D image based on a right eye position; and obtaining the side-by-side 3D image by coloring the left mesh area and the right mesh area projected onto the coordinate system of the side-by-side 3D image based on the raw image.

Description

Method and computer device for generating side-by-side three-dimensional images

The present invention relates generally to image processing techniques, and in particular to the generation of side-by-side three-dimensional (3D) images.

Side-by-side 3D images are composed of two images aligned side by side. The two images appear to be the same, but are actually slightly different, simulating the two images we see with our left and right eyes respectively. The difference between the two images is integrated and processed by the brain, allowing the viewer to perceive a 3D effect.

Side-by-side 3D images are usually displayed on wearable devices, such as 3D/VR glasses or head-mounted displays. In order to display side-by-side 3D images on a naked-eye stereoscopic display device, more technologies in the image rendering pipeline are required, such as eye tracking and image weaving (or interleaving). In some application scenarios, such as virtual reality (VR) scenes or stereoscopic 3D games, the user's position (and of course the user's eyes) may change frequently and rapidly. Whenever the user's position changes, the screen to be displayed needs to be recalculated, which consumes considerable computing resources.

In view of the above problems, it is hoped that there will be a method and computer device that can generate side-by-side 3D images more efficiently than traditional methods.

One embodiment of the present disclosure provides a method for generating side-by-side three-dimensional (3D) images based on original images. The method is performed by a computer device. The method includes the following steps: create a 3D grid; update the left grid area and right grid area of the 3D grid based on the estimated depth information of the original image; update each grid in the left grid area based on the left eye position. Project the vertices onto the coordinate system of the side-by-side 3D image, and project each grid vertex in the right grid area onto the coordinate system of the side-by-side 3D image based on the right eye position; and based on the original image, project the coordinates on the side-by-side 3D image The left and right grid areas on the system are shaded to obtain side-by-side 3D images.

In some embodiments, the step of creating a 3D mesh includes creating a texture of a triangle formed by three adjacent mesh vertices.

In some embodiments, the step of coloring the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image based on the original image to obtain the side-by-side 3D image includes interpolating the color texture of the original image. Capture (interpolate) the left and right grid areas projected on the coordinate system of the side-by-side 3D image.

One embodiment of the present disclosure provides a computer device for generating side-by-side 3D images based on original images. The computer device includes a central processing unit (CPU) and a graphics processing unit (GPU). The CPU is set to run the program to request the GPU to perform the steps of the above method.

The disclosed method and computer device provide a novel image processing technique that generates side-by-side images by simultaneously rendering the left and right views on the same grid, instead of rendering the left and right eyes twice separately and then aligning them. practice. By only loading resources (for example, mesh models, depth information, color textures, etc.) into the GPU once, CPU calls can be saved. In addition, the possibility of synchronization issues between CPU calls and hardware execution is reduced.

The following description enumerates various embodiments of the present invention, but is not intended to limit the content of the present invention. The actual scope of the invention is defined by the scope of the patent application.

In each of the embodiments listed below, the same or similar elements or components will be represented by the same reference numerals.

It should be understood that the terms "include" and "include" used in the specification indicate the existence of specific technical features, values, method steps, program operations, elements and/or components, but do not exclude additional technical features, values, Method steps, program operations, elements, components, or any combination of the above.

The serial numbers in this specification and the scope of the patent application, such as "first", "second", etc., are only for convenience of explanation, and there is no sequential relationship between them.

FIG. 1 is a schematic diagram illustrating a conceptual flow of an example conventional approach to generating side-by-side 3D images 120 . In the example shown, a mesh model is used to define the shape and depth of objects in the image (eg, mountains).

As shown in Figure 1, the created mesh 101 includes a plurality of triangles (can be replaced by other types of polygons) arranged sequentially on the X-Y plane, and each triangle has a set of vertices. By giving the vertices of each triangle in mesh 101 a specific height (ie, a scale along the Z-axis direction), mesh 101 is updated to updated mesh 102. The specific height is determined by the depth information 100 captured from the original image and stored in the depth buffer (also known as the "z-buffer"). Then, by shading the triangles in the updated mesh 102, a left eye image 103 (ie, the view of the human left eye) is generated. Similarly, the created grid 111 includes a plurality of triangles (can be replaced by other types of polygons) arranged sequentially on the X-Y plane, each triangle having a set of vertices. By giving the vertices of each triangle in mesh 111 a specific height (ie, a scale along the Z-axis direction), mesh 111 is updated to updated mesh 112 . This specific height is determined by the depth information 100 captured from the original image and stored in the depth buffer. Then, a right-eye image 113 (ie, the view of the human right eye) is generated by shading the triangles in the updated mesh 112 based on the color texture of the original image 10 . The left eye image 103 and the right eye image 113 appear to be the same, but are actually slightly different. Finally, a single side-by-side 3D image 120 is generated by aligning the left-eye image 103 and the right-eye image 113 side by side. The left part of the side-by-side 3D image 120 is from the left-eye image 103, and the right part of the side-by-side 3D image 120 is from the right-eye image 113.

As can be seen in Figure 1, the side-by-side 3D image 120 is generated by drawing the screen twice, once for each eye, which means that some of the resources used to draw the side-by-side 3D image 120 will only draw the left eye image 103 Or just draw the right eye image twice as much as 113. For example, grid 101 and grid 111 are created using the same grid model, so the grid model is loaded from memory twice. The depth information 100 of the original image 10 required to obtain the updated meshes 102 and 112 is the same, but it is loaded from the depth buffer twice, once for each of the updated meshes 102 and 112. The color texture (for example, the RGB values of each pixel of the original image 10 required to obtain the left-eye image 103 and the right-eye image 113) is also loaded from memory twice to update the grid 102 and 112 once each.

FIG. 2 is a schematic diagram illustrating a conceptual process of generating side-by-side 3D images 203 according to an embodiment of the present disclosure. In this example, grid 201 is a larger grid created with twice the width of grids 101 and 111 . Update the grid 201 to the depth information 100 of the original image 10 required for the updated grid 202, and obtain the updated grid 102 Or 112 the required depth information 100 of the original image 10 is the same. The left and right parts of grid 202 have been updated, respectively for the left eye and the right eye. Therefore, side-by-side 3D images 203 can be obtained by directly shading the left and right portions of a single updated mesh 202, without the need to draw the left and right eye images separately and then align them. In this way, resources such as the mesh model, depth information 100 and color texture of the original image 10 are only loaded once instead of twice.

FIG. 3 is a schematic block diagram of a computer device 300 for generating side-by-side three-dimensional (3D) images according to an embodiment of the present disclosure. As shown in FIG. 3 , the computer device 300 includes components such as a central processing unit (CPU) 301 , a graphics processing unit (GPU) 302 , an input device 303 and a storage device 304 , which are coupled to each other through a system bus 305 .

The computer device 300 is an electronic device capable of performing computing tasks, such as a personal computer (including a desktop computer, a notebook computer, and a tablet computer), or a server computer.

The CPU 301 is a general-purpose processor configured to run graphics drivers 307 and application programs 308 stored in the storage device 304 . According to an embodiment of the present disclosure, the CPU 301 is further configured to run a first program 309 stored in the storage device 304 to request the GPU 302 to execute steps of a method for generating side-by-side three-dimensional (3D) images. The steps of this method will be described later with reference to Figure 4.

The GPU 302 may be an electronic circuit specially designed to perform computer graphics operations and image processing to reduce the burden of a general central processing unit (CPU). Therefore, the GPU 302 is used for graphics operations and image processing, and is more powerful than the general-purpose CPU 301. efficiency.

The input device 303 may include a keyboard, a mouse, a touch panel, or any other device capable of receiving control commands from the viewer, but the invention is not limited thereto. In one embodiment, the input device 303 allows the viewer to enable/disable the 3D image, in other words, to switch between the 2D mode and the 3D mode of the display. In another embodiment, the input device 303 allows the viewer to adjust (increase or decrease) interpupillary distance (IPD) parameter settings, which relate to how the 3D grid is projected onto the coordinate system of the side-by-side 3D image.

The storage device 304 may be a non-volatile storage device, such as a hard disk, a solid state drive (SSD), a flash memory or a read-only memory, but the present disclosure is not limited thereto. According to the embodiment of the present invention, the storage device 304 is used to store the graphics driver 307, the application program 308 and the first program 309.

The graphics driver 307 is a set of software programs that allows the operating system (such as Windows, Linux, MacOS, etc.) with the application 308 installed to communicate with the GPU 302 .

The application program 308 can be any software program that provides visual images or pictures to viewers, such as games, audio/video/multimedia player programs, and web browsers (for example, when the viewer uses it to browse online video platforms such as YouTube or Twitter) when watching videos on social media), photo viewing apps, or other visual entertainment apps.

In the example scenario in Figure 3, since the display device presents the left-eye image and the right-eye image (i.e., the first image and the second image as mentioned above) on the same screen, The naked-view stereoscopic display device 310 is configured so that the GPU 302 is further configured to execute an interlacing (or weaving) process based on the side-by-side images (such as the side-by-side images 120 in Figure 1 and the side-by-side images 203 in Figure 2) to generate a stereoscopic image. After generating the stereoscopic image, the GPU 302 transmits the stereoscopic image to the naked-eye stereoscopic display device 310 through the transmission interface 306 .

The transmission interface 306 may include a wired transmission interface and/or a wireless transmission interface. Wired transmission interfaces may include High Definition Multimedia Interface (HDMI), DisplayPort (DP) interface, embedded DisplayPort (eDP) interface, Universal Serial Bus (USB) interface, USB Type-C interface, Thunderbolt interface, Digital Video Interface (DVI) or a combination thereof. The wireless transmission interface may include fifth-generation (5G) wireless systems, Bluetooth, WiFi, near field communication (Near Field Communication; NFC) interfaces, etc., but this disclosure is not limited to this.

The naked-view stereoscopic display device 310 is a display device that allows viewers to display stereoscopic images without using a wearable device (such as 3D/VR glasses or a helmet). The naked-eye stereoscopic display device 310 may include a display controller 311 and a display panel 312 . The display controller 311 is used to switch the display mode of the display panel 312 according to the display mode control signal from the computer device 300 (for example, as described above, allowing the viewer to switch between the 2D mode and the 3D mode through the input device 303. Display The panel 312 can use any suitable autostereoscopic technology in the technical field of the disclosure to achieve the effect of autostereoscopic vision, such as parallax barrier, lenticular lenses, directional backlight, integral imaging ( integral imaging)...etc., which will not be described again here. Thus, in response to the naked-view stereoscopic display device 310 After receiving the stereoscopic image, the stereoscopic 3D image of the application can be presented to the viewer by displaying the stereoscopic image on the display panel.

Figure 4 is a flowchart of a method 400 for generating side-by-side three-dimensional (3D) images based on original images according to an embodiment of the present disclosure. The method 400 may be executed by a computer device, such as the computer device 300 in Figure 3 . The CPU 301 of the computer device 300 is used to run the first program 309 to request the GPU 302 to execute the steps of the method 400. As shown in Figure 4, method 400 may include steps 401-405.

In step 401, a 3D mesh (such as mesh 201 in Figure 2) is created. Method 400 then proceeds to step 402.

In step 402, the depth information of the original image is estimated. Method 400 then proceeds to step 403.

In step 403, the left grid area and the right grid area of the 3D grid are updated based on the estimated depth information of the original image. Method 400 then proceeds to step 404.

In step 404, each grid vertex of the left grid area is projected onto the coordinate system of the side-by-side 3D image based on the left eye position (i.e., the position of the viewer's left eye), and based on the right eye position (i.e., the viewer's right eye position) projects each grid vertex in the right grid area onto the coordinate system of the side-by-side 3D image. Method 400 then proceeds to step 405.

In step 405, based on the original image, the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image are shaded to obtain the side-by-side 3D image.

In some embodiments, step 401 includes the operation of creating a texture of a triangle formed by three mesh vertices adjacent to each other. For example, as shown in Figure 2, the mesh 201 includes a plurality of vertices and a plurality of triangles. Each triangle is composed of three adjacent mesh vertices and has corresponding textures. When the mesh is created, a texture for each triangle is also created (or initialized) as a data structure.

In further embodiments, the depth information may be represented in the form of a depth map, a parallax map, or a point cloud, but the present disclosure is not limited thereto. A model based on a convolutional neural network (CNN) may be used to estimate depth information in step 402, but the present disclosure is not limited to this.

In some embodiments, the left eye position and the right eye position used in step 404 can be measured using any known eye tracking technology, and this disclosure is not limited thereto.

In some embodiments, step 405 may further include an operation of interpolating the color texture of the original image into the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image. The color texture can be expressed in a variety of color space formats, such as RGB, YUV, YCbCr, CMYK, sRGB, HSV, etc., and this disclosure is not limited to this.

In some embodiments, a stereoscopic image can be generated based on the side-by-side 3D images obtained in step 405 by performing an interlacing (or weaving) process. Then, the stereoscopic image is sent to a naked-eye stereoscopic display device to display the stereoscopic image.

The steps of the methods and algorithms provided in this disclosure can be directly applied to hardware and software modules or a combination thereof by executing the processor. Software modules (including execution instructions and related data) and other data can be stored in data memory, such as random access memory (RAM), flash memory, read-only memory (ROM), erasable and Programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), scratchpad, hard drive, mobile hard drive, CD-ROM, DVD, or any other computer-readable device in the field Storage media format. For example, the storage medium may be coupled to a machine device, such as a computer/processor (for convenience of description, represented by a processor in this disclosure). The processor can read information (such as program code) from the storage medium and can write information to the storage medium. The storage medium can be integrated with the processor. Application Specific Integrated Circuits (ASICs) include processors and storage media. The observation device contains an ASIC. In other words, the processor and storage medium are included in the observation device rather than directly connected to the observation device. Additionally, in some embodiments, any computer program-enabled product includes a readable storage medium, where the storage medium contains program code related to one or more disclosed embodiments. In some embodiments, a computer program product may include packaging materials.

The disclosed method and computer device provide a novel image processing technique that generates side-by-side images by simultaneously rendering the left and right views on the same grid, instead of rendering the left and right eyes twice separately and then aligning them. practice. By only loading resources (for example, mesh models, depth information, color textures, etc.) into the GPU once, CPU calls can be saved. In addition, the possibility of synchronization issues between CPU calls and hardware execution is reduced.

The above paragraphs are narrated in various ways. Obviously, the teachings herein may be implemented in a variety of ways, and any specific architecture or functionality disclosed in the examples is only a representative case. Based on the teachings of this article, it should be understood in the art that each aspect disclosed in this article can be implemented independently, or two or more aspects can be combined and implemented.

Although the present disclosure has been described above using embodiments, they are not intended to limit the present disclosure. Anyone skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the invention is The scope of the patent application shall be determined by the attached patent application.

10:Original image 100:In-depth information 101:Grid 102: Grid updated 103:Left eye image 111:Grid 112: Grid updated 113:Right eye image 120: Side-by-side 3D images 201:Grid 202: Grid updated 203: Side-by-side 3D images 300:Computer device 301:CPU 302:GPU 303:Input device 304:Storage device 305: System bus 306:Transmission interface 307: Graphics driver 308:Application 309:First program 310: Naked stereoscopic display device 311:Display controller 312:Display panel 400:Method 401-405: Steps

The present disclosure will be better understood from the description of the following exemplary embodiments together with the accompanying drawings. In addition, it should be understood that in the flowchart of the present disclosure, the execution order of each block may be changed, and/or some blocks may be changed, deleted, or merged. Figure 1 is a schematic diagram illustrating a conceptual flow of an example conventional approach to generating side-by-side 3D images. FIG. 2 is a schematic diagram illustrating a conceptual process of generating side-by-side 3D images according to an embodiment of the present disclosure. FIG. 3 is a schematic block diagram of a computer device for generating side-by-side three-dimensional (3D) images according to an embodiment of the present disclosure. FIG. 4 is a flowchart of a method for generating side-by-side three-dimensional (3D) images based on original images according to an embodiment of the present disclosure.

400:Method

401-405: Steps

Claims (8)

A method for generating side-by-side 3D images. The side-by-side 3D images are generated based on an original image. The method is executed by a computer device and includes the following steps: creating a 3D mesh, wherein The 3D grid includes a left grid area and a right grid area, and the left grid area and the right grid area each have a plurality of grid vertices; the depth information of the original image is estimated; based on the estimated The depth information of the original image, updating the left grid area and the right grid area of the 3D grid; projecting each grid vertex of the left grid area to one of the side-by-side 3D images based on a left eye position on the coordinate system, and project each grid vertex of the right grid area onto the coordinate system of the side-by-side 3D image based on a right eye position; and based on the original image, project the coordinates on the side-by-side 3D image The left grid area and the right grid area on the system are shaded to obtain the side-by-side 3D image. As in the method of claim 1, the step of creating the 3D mesh includes: creating a triangle texture composed of three adjacent vertices of the mesh. The method of claim 1, wherein the step of shading the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image based on the original image to obtain the side-by-side 3D image includes: : Interpolate the color texture of the original image into the projected side-by-side 3D The left grid area and the right grid area on the coordinate system of the image. The method of claim 1 further includes: generating a stereoscopic image based on the side-by-side 3D image, and sending the stereoscopic image to a naked-eye stereoscopic display device to display the stereoscopic image. A computer device that generates side-by-side 3D images. The side-by-side 3D images are generated based on an original image. The computer device includes a central processing unit (CPU) and a graphics processing unit (GPU), wherein the The CPU is configured to run a program to request the GPU to perform the following steps: create a 3D mesh (mesh), wherein the 3D mesh includes a left mesh area and a right mesh area, and the left mesh area and the right mesh area Each grid area has a plurality of grid vertices; estimate the depth information of the original image; update the left grid area and the right grid area of the 3D grid based on the estimated depth information of the original image; based on A left eye position projects each grid vertex of the left grid area onto a coordinate system of the side-by-side 3D image, and a right eye position projects each grid vertex of the right grid area onto the side-by-side on the coordinate system of the 3D image; and based on the original image, coloring the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image to obtain the side-by-side 3D image. The computer device of claim 5, wherein the step of creating the 3D mesh includes: creating a triangle texture composed of three adjacent vertices of the mesh. The computer device of claim 5, wherein the step of shading the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image based on the original image to obtain the side-by-side 3D image, include: Interpolate the color texture of the original image into the left grid area and the right grid area projected on the coordinate system of the side-by-side 3D image. The computer device of claim 5, wherein the CPU is further configured to request the GPU to generate a stereoscopic image based on the side-by-side 3D image, and send the stereoscopic image to a naked-eye stereoscopic display device to display the stereoscopic image.