WO2025225483A1 - Image processing device, image processing method, and program - Google Patents

Abstract

Provided is an image processing device that acquires a frame image constituting a video to be presented to a user each time the frame image is drawn, and outputs a converted frame image obtained by converting the acquired frame image using a machine learning model prepared in advance as the frame image to be presented to the user.

Description

Image processing device, image processing method, and program

The present invention relates to an image processing device, an image processing method, and a program for generating images to be displayed on a display device.

For example, in video games, an application program such as a game program draws frame images in real time, outputs the drawn frame images as video signals, and repeatedly executes a process to display them on the screen of a display device. In this way, images are presented to the user.

In the conventional technologies described above, the frame images drawn by the application program may not match the user's viewing environment or preferences in terms of resolution, color tone, etc. For this reason, technologies have been known that use hardware logic or other methods to convert the resolution, etc., of frame images before outputting them. However, because such technologies perform fixed conversions, they are not necessarily suitable for the content of the image.

The present invention was made in consideration of the above situation, and one of its objectives is to provide an image processing device, image processing method, and program that can present to the user the results of desired conversions made according to the content of a frame image.

An image processing device according to one aspect of the present invention includes a frame image acquisition unit that acquires a frame image each time the frame image constituting a video to be presented to a user is drawn, and a frame image conversion unit that converts the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputs the converted frame image as the frame image to be presented to the user.

An image processing method according to one aspect of the present invention includes the steps of acquiring a frame image each time the frame image constituting a video to be presented to a user is drawn, and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user.

A program according to one aspect of the present invention causes a computer to execute the following steps: acquiring a frame image each time the frame image constituting a video to be presented to a user is drawn; and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user. This program may be provided by being stored on a computer-readable, non-transitory information storage medium.

1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention; 1 is a functional block diagram showing functions of an image processing device according to an embodiment of the present invention; FIG. 3 is a timing chart showing an example of a flow of processing executed by the image processing device according to the embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an image processing device 10 according to one embodiment of the present invention. The image processing device 10 is, for example, a home game console or a personal computer, and as shown in the diagram, is configured to include a control unit 11, a storage unit 12, and an interface unit 13. The image processing device 10 is also connected to a display device 14 and an operation device 15.

The control unit 11 includes at least one processor such as a CPU, and executes programs stored in the storage unit 12 to perform various information processing. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores the programs executed by the control unit 11 and the data processed by those programs.

The interface unit 13 is an interface for data communication between the display device 14 and the operation device 15. The image processing device 10 is connected to the display device 14 and the operation device 15 via the interface unit 13, either wired or wirelessly. Specifically, the interface unit 13 includes a multimedia interface for transmitting video signals supplied by the image processing device 10 to the display device 14. It also includes a data communication interface for receiving signals indicating operations performed by the user on the operation device 15. Furthermore, the interface unit 13 may be equipped with a communication interface for sending and receiving data to and from other communication devices via a communication network such as the Internet.

The display device 14 displays on a screen an image corresponding to the video signal supplied from the image processing device 10. The display device 14 may be a stationary display device such as a home television set, or a portable display device. The display device 14 may also be a head-mounted display device capable of presenting a three-dimensional image by presenting separate images to each of the user's left and right eyes.

The operation device 15 is, for example, a controller for a home game console, and accepts operation input from the user. The operation device 15 is connected to the image processing device 10 via a wired or wireless connection, and transmits operation signals indicating the content of the operation input accepted from the user to the image processing device 10. The operation device 15 may be of various shapes, such as a device that the user holds in their hand or a device that the user wears on their hand.

The functions realized by the image processing device 10 will be explained below using the functional block diagram of Figure 2. As shown in Figure 2, the image processing device 10 is functionally configured to include an application execution unit 21, a frame image acquisition unit 22, and a frame image conversion unit 23. These functions are realized by the control unit 11 operating in accordance with one or more programs stored in the storage unit 12. These programs may be provided to the image processing device 10 via a communications network such as the Internet, or may be provided by being stored on a computer-readable information storage medium such as an optical disc.

The application execution unit 21 executes an application program and repeatedly performs the process of drawing frame images that show the results of that processing. These frame images are images that make up the video to be presented to the user, and by displaying such frame images on the display device 14 at predetermined intervals, the user can view the video that is the result of processing by the application execution unit 21. Note that the drawing of frame images may be performed by a processor such as a GPU based on drawing commands from the application program. The drawn frame images are written to a predetermined frame buffer memory allocated in the storage unit 12.

The frame image acquisition unit 22 reads and acquires the drawn frame image from the frame buffer memory each time the application execution unit 21 draws a frame image. In other words, in this embodiment, the frame image drawn by the application execution unit 21 is not sent directly to the display device 14.

Each time the frame image acquisition unit 22 acquires a newly drawn frame image, the frame image conversion unit 23 performs a given conversion process on the acquired frame image. This conversion process is a process for adjusting the content of the frame image according to the user's viewing environment, etc. For ease of explanation, the frame image obtained by the conversion process by the frame image conversion unit 23 will be referred to as the converted frame image, and the frame image acquired by the frame image acquisition unit 22 before the conversion process is performed will be referred to as the pre-conversion frame image. In this embodiment, this converted frame image is output as the frame image to be presented to the user. The converted frame image output by the frame image conversion unit 23 is sent to the display device 14 via the interface unit 13 and displayed on the screen of the display device 14.

Here, the frame image conversion unit 23 converts the frame images using a pre-prepared machine learning model. This allows conversion to be performed according to the content of each frame image, compared to conversion using hardware logic, fixed filters, etc.

In this embodiment, multiple machine learning models are prepared in advance depending on the desired conversion content, etc. The frame image conversion unit 23 performs the frame image conversion process using a machine learning model selected from these multiple machine learning models based on given conditions such as user selection. The types of machine learning models prepared, the selection criteria for selecting the machine learning model to use, and specific examples of how to generate a machine learning model will be described later.

Here, an example of the flow of processing executed by the image processing device 10 according to this embodiment will be explained using the timing diagram in Figure 3. Note that in the diagram, (N-1), (N), and (N+1) denote numbers conveniently assigned to the frame images that are drawn in sequence.

As shown in the figure, the application execution unit 21 begins drawing a new pre-conversion frame image every time a predetermined time interval Tf has elapsed. This time interval Tf is determined according to the frame rate; for example, if the frame rate is 60 fps, the time interval Tf is 1/60 seconds. Note that in this embodiment, the drawing process of the pre-conversion frame image is expected to be completed in a time shorter than the time interval Tf. In the figure, the drawing process of the pre-conversion frame image (N) begins at time t0, and the drawing process of the pre-conversion frame image (N+1) begins at time t1, after the time interval Tf has elapsed since time t0.

Once the drawing process for the pre-conversion frame image is complete, the frame image conversion unit 23 performs conversion processing on the pre-conversion frame image and outputs a post-conversion frame image. This conversion processing, together with the drawing process for the pre-conversion frame image, is expected to be completed within the time interval Tf.

The converted frame image output by the frame image conversion unit 23 is transmitted to the display device 14 in the next cycle. In other words, when a new cycle begins and the drawing process for the next frame image to be displayed begins, the converted frame image output in the previous cycle is transmitted to the display device 14. In the figure, the converted frame image (N-1) whose conversion was completed by time t0 is transmitted to the display device 14 between time t0 and time t1, the converted frame image (N) whose conversion was completed by time t1 is transmitted to the display device 14 between time t1 and time t2, and the converted frame image (N+1) whose conversion was completed by time t2 is transmitted after time t2. By repeating this cycle, the image processing device 10 can perform a given conversion process on all pre-conversion frame images in real time and display the resulting converted frame images on the display device 14.

Below, we will explain some specific examples of the frame image conversion process performed by the frame image conversion unit 23.

As a first example, we will explain an example of conversion processing to increase the resolution of frame images. In recent years, display devices capable of displaying at extremely high resolutions such as 4K have appeared, but older application programs, for example, may not be able to display at such high resolutions. In such cases, by converting pre-conversion frame images drawn by the older application program into high-resolution post-conversion frame images, it is possible to present the user with high-quality video that can be displayed by the display device 14.

Specifically, the frame image conversion unit 23 converts the pre-conversion frame image into a post-conversion frame image with a higher, predetermined resolution. In this case, by utilizing a machine learning model, the frame image conversion unit 23 is able to not only increase the number of pixels in the post-conversion frame image, but also to increase the resolution of each pixel in the pre-conversion frame image by taking into account the content of the surrounding pixels.

In this example, in order to support multiple types of resolution, multiple machine learning models with different output resolutions may be prepared in advance. The frame image conversion unit 23 performs the frame image conversion process using a machine learning model from the multiple machine learning models prepared in advance that converts the frame image to one with a resolution that can be displayed by the display device 14. More specifically, for example, the frame image conversion unit 23 selects, as the machine learning model to use, a machine learning model that converts the image to a resolution that the user has specified in advance on a settings screen or the like. Alternatively, the frame image conversion unit 23 may obtain information regarding the display performance of the display device 14, and based on the obtained information, select, for example, the maximum resolution that the display device 14 can display as the resolution of the converted frame image. This makes it possible to display images at a variety of resolutions.

As a second example, we will explain how to adjust the brightness and color gamut of pixels contained in a frame image. As with the resolution in the first example, the color gamut that a display device can display also differs depending on the type of display device. Therefore, by converting a pre-conversion frame image drawn by an application program that does not support display in a wide color gamut into a post-conversion frame image with a wider color gamut, it is possible to present the user with high-quality images that can be displayed by the display device 14.

Specifically, the frame image conversion unit 23 converts the colors of the pixels contained in the frame image so that the color gamut used in the converted frame image is wider than the color gamut used in the pre-conversion frame image. For example, the frame image conversion unit 23 converts a pre-conversion frame image with a dynamic range of SDR (Standard Dynamic Range) into a post-conversion frame image with a dynamic range of HDR (High Dynamic Range).

In this example as well, by using a machine learning model, the color of each pixel in the converted frame image is determined taking into account the colors of surrounding pixels, etc. This makes it possible to convert parts that cannot be fully expressed in SDR and are crushed into colors with a gradation that looks natural to the human eye. Also in this example as well, multiple machine learning models may be prepared in advance to suit various color gamuts. The frame image conversion unit 23 may then select the machine learning model to use based on user specifications, information related to the display performance of the display device 14, etc.

Furthermore, regardless of the color gamut used, depending on the drawing conditions, such as the color space used when drawing the pre-conversion frame image, subtle color changes may not be expressed within the frame image, resulting in crushed colors. Therefore, the frame image conversion unit 23 may perform a conversion that changes the brightness of each pixel contained in the pre-conversion frame image according to the brightness of the surrounding pixels, while maintaining the possible range of brightness before and after conversion. With this type of conversion, by increasing or decreasing the brightness of some pixels, it is possible to make the brightness change overall within the converted frame image in a gradation that appears natural to the human eye.

As a third example, we will explain an example of conversion processing that converts the colors of a frame image to a color tone that matches the viewer's color vision characteristics. Color vision diversity is known as the difference in how different people see colors. Due to this characteristic, some people may find it difficult to distinguish between certain colors. Therefore, in this example, the colors of the pixels contained in the pre-conversion frame image are converted to colors that are easy to see for viewers with specific color vision characteristics.

In this example, simply converting a specific color to another color mechanically will not make the image easier for the viewer to see. For example, for people with color vision disorders who have difficulty distinguishing between two specific colors, when those two colors are adjacent, the colors need to be converted so that they are easier to distinguish. Therefore, it is desirable to decide what color to convert the color of each pixel into by taking into account the color composition and distribution of the image itself, and the color arrangement of the pixels surrounding each pixel.

In this example, the frame image conversion unit 23 uses a machine learning model that has been prepared in advance according to the type of color vision characteristics of the viewer to convert the colors of the pixels contained in the pre-conversion frame image to a color tone that corresponds to the color vision characteristics of the viewer. This makes it possible to perform color conversion that takes into account the content of the frame image.

In this example, it is assumed that multiple machine learning models are prepared in advance according to various types of color vision characteristics. The user of the image processing device 10 selects one machine learning model in advance, according to their own color vision characteristics, on a settings screen or the like. The frame image conversion unit 23 converts the frame image using the machine learning model selected by the user. This allows the user to always view images with color tones that are easy to see according to their own color vision characteristics, regardless of the type of application program that draws the frame image or the content of the image to be displayed.

The frame image conversion unit 23 may also perform a conversion that combines multiple examples described above. As a specific example, the frame image conversion unit 23 may first perform a conversion that changes the color tone of the pre-conversion frame image, and then perform a conversion that further improves the resolution of the frame image whose color tone has been changed, thereby generating a post-conversion frame image.

Furthermore, machine learning models for performing multiple types of conversion at once may be prepared in advance. In this case, machine learning models generated by performing machine learning individually for each combination of conversions expected to be required are prepared in advance. As an example, if high-resolution conversion corresponding to two types of output resolution and color tone conversion corresponding to two types of color vision characteristics are required, a total of nine types of conversion combinations are expected (three types x three types), taking into account the possibility that only one of the conversions is required. For one of these, it is sufficient to output the pre-conversion frame image as is without performing either conversion, so machine learning models are prepared for each of the remaining eight types of conversion combinations. The user selects one of these eight machine learning models to use based on their own color vision characteristics and the supported resolution of the display device 14. By converting the frame image using this selected machine learning model, it is possible to obtain a converted frame image with the color tone and resolution desired by the user in a single conversion.

Furthermore, the frame image conversion unit 23 may select a machine learning model to use depending on the type of application program that draws the pre-conversion frame image, and convert the frame image using the selected machine learning model. For example, if the application program is a game program, different machine learning models may be prepared in advance for each game genre or title. By converting the frame image using a machine learning model selected depending on the type of application program in this way, it is possible to perform conversion that corresponds to the tendencies of the frame image being drawn, and to make the color tone of the converted frame image suitable for the content of the application program.

The multiple machine learning models corresponding to the multiple types of conversion described above may be stored in advance in the storage unit 12 of the image processing device 10, or may be provided to the image processing device 10 as needed via a communications network, etc. For example, a predetermined server device connected to the image processing device 10 via the Internet may store in advance multiple machine learning models corresponding to various types of conversion, and may transmit the model data necessary to execute the machine learning models in response to a request from the image processing device 10. In this case, when a user of the image processing device 10 selects a conversion content on a settings screen, etc., the image processing device 10 requests model data of the machine learning model corresponding to the selected conversion content from the server device. The machine learning model provided by the server device in response to this request is then stored in the storage unit 12, and the frame image is converted using that model. This makes it easy to update the machine learning model over time and improve the quality of the conversion process.

Below, we will explain a specific example of how to generate the machine learning model used to convert the frame images described above.

The machine learning model used by the frame image conversion unit 23 may be a model obtained by machine learning using two frame images obtained by drawing the same content under different drawing conditions, one as input data and the other as training data.

As a specific example, when the frame image conversion unit 23 converts a frame image to a higher resolution, it first has an application program that supports both low and high resolutions execute the same process multiple times while changing the output resolution. This results in high-resolution and low-resolution frame images that represent the same content being drawn. Of these, machine learning is performed using the low-resolution frame images as input data and the high-resolution frame images as training data to generate a machine learning model that can convert low-resolution frame images into high-resolution frame images. In this way, there is no need to prepare manually adjusted images, and a machine learning model that can increase the resolution of frame images can be efficiently generated.

Furthermore, when the frame image conversion unit 23 converts the color tone of a frame image, it causes an application program equipped with an output mode that accommodates color vision diversity to execute the same process multiple times in different output modes. This makes it possible to render frame images with the same content, with unadjusted color tones and frame images with color tones adjusted for people with specific color vision characteristics. Of these, by performing machine learning using frame images with unadjusted color tones as input data and adjusted frame images as training data, it is possible to efficiently generate a machine learning model that can convert colors to those that are easier to view for people with specific color vision characteristics.

Furthermore, the machine learning model used by the frame image conversion unit 23 may be a model obtained by machine learning using, as input data, images obtained by performing a given conversion on images used as training data.

As a specific example, if a frame image drawn by an application program that supports high resolution is used as training data, machine learning can be performed using the image obtained by converting this frame image to a lower resolution as input data. This makes it possible to obtain a machine learning model that can restore the original image by performing the inverse conversion, which is to increase the resolution. Similarly, if you want to perform a conversion to expand the color gamut, machine learning can be performed using a frame image drawn with a wide brightness range, such as HDR, as training data, and the image obtained by converting this frame image to a narrower brightness range, such as SDR, as input data. This makes it possible to generate a machine learning model that can convert to a wide color gamut.

In the above explanation, a single frame image is used as input to the machine learning model, but the frame image conversion unit 23 may also convert frame images using a machine learning model that accepts multiple frame images as input. In particular, by using a machine learning model that accepts multiple frame images drawn consecutively in chronological order as input, it is possible to convert the most recent pre-conversion frame image into a post-conversion frame image, taking into account changes over time in the content of the frame images.

As described above, the image processing device 10 according to this embodiment can effectively convert the contents of a frame image and present the converted frame image to the user in real time.

Note that embodiments of the present invention are not limited to those described above. For example, the content of the conversion process performed by the frame image conversion unit 23 is not limited to the examples given above, and various types of conversion may be performed on the frame images.

Furthermore, in the above explanation, the image processing device 10 is an information processing device that is located relatively close to the user and is directly connected to the display device 14 and operation device 15. However, this is not limited to this. For example, in services such as cloud gaming services, a server device connected to the client device via a communications network may render frame images to be displayed on the screen of the display device 14, rather than a client device directly connected to the display device 14 and operation device 15 used by the user. In such cases, the server device connected to the client device used by the user via a communications network may function as the image processing device 10 of the present invention. In this case, the server device functioning as the image processing device 10 executes an application program to convert the rendered pre-conversion frame images to generate converted frame images, and transmits the generated converted frame images to the client device.

Furthermore, in the above explanation, the image processing device 10 also executes an application program that draws pre-conversion frame images, but this is not limited to this. The image processing device 10 according to an embodiment of the present invention may also acquire pre-conversion frame images drawn by another information processing device, convert them into post-conversion frame images, and send them to the display device 14 for viewing by the user. For example, in the cloud gaming service mentioned above, a server device may draw pre-conversion frame images, send the drawn pre-conversion frame images to a client device functioning as the image processing device 10, and convert the received pre-conversion frame images into post-conversion frame images that are presented to the user.

It should be noted that the functions provided by the components described herein may be implemented by any circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and/or combinations thereof, configured or programmed to provide the described functions. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory.

In this specification, a circuit, unit, or means is hardware that is programmed to realize a described function or that performs that function. The hardware may be any hardware disclosed in this specification or any hardware that is programmed to realize or known to perform the described function. In the case where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software used to operate the hardware and/or processor.

The present disclosure may include the following aspects.
[Item 1]
Each time a frame image constituting a video to be presented to a user is drawn, the frame image is acquired;
converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user;
An image processing device comprising a circuit configured as follows.
[Item 2]
Item 1. In the image processing device according to item 1,
The circuit performs a conversion to increase the resolution of the acquired frame image.
Image processing device.
[Item 3]
Item 1. In the image processing device according to item 1,
the circuit performs a conversion to change the brightness of pixels included in the acquired frame image in accordance with the brightness of surrounding pixels;
Image processing device.
[Item 4]
Item 1. In the image processing device according to item 1,
the circuit converts the colors of pixels included in the acquired frame image so that a color gamut used in the converted frame image is wider than a color gamut used in the acquired frame image;
Image processing device.
[Item 5]
Item 1. In the image processing device according to item 1,
the circuit converts the color of pixels included in the acquired frame image into a color tone according to the color vision characteristics of the viewer;
Image processing device.
[Item 6]
Item 1. In the image processing device according to item 1,
the circuit performs the conversion using a machine learning model selected based on a given condition from a plurality of machine learning models prepared in advance;
Image processing device.
[Item 7]
Item 6. In the image processing device according to item 6,
the circuit performs the conversion using a machine learning model selected according to the type of application program that has rendered the acquired frame image;
Image processing device.
[Item 8]
Item 1. In the image processing device according to item 1,
The machine learning model is a model obtained by machine learning using two frame images obtained by drawing the same content under different drawing conditions, one of which is used as input data and the other as training data.
Image processing device.
[Item 9]
Each time a frame image constituting a video to be presented to a user is drawn, the frame image is acquired;
converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user;
Image processing methods.
[Item 10]
A computer-readable, non-transitory information storage medium, comprising:
Each time a frame image constituting a video to be presented to a user is drawn, the frame image is acquired;
converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user;
An information storage medium that stores a program for causing a computer to execute a process.

10 Image processing device, 11 Control unit, 12 Memory unit, 13 Interface unit, 14 Display device, 15 Operation device, 21 Application execution unit, 22 Frame image acquisition unit, 23 Frame image conversion unit.

Claims (10)

a frame image acquisition unit that acquires a frame image constituting a video to be presented to a user each time the frame image is drawn;
a frame image conversion unit that converts the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputs the converted frame image as a frame image to be presented to the user;
An image processing device comprising: 2. The image processing device according to claim 1,
the frame image conversion unit converts the acquired frame images to a higher resolution;
Image processing device. 2. The image processing device according to claim 1,
the frame image conversion unit performs conversion to change the luminance of pixels included in the acquired frame image in accordance with the luminance of surrounding pixels;
Image processing device. 2. The image processing device according to claim 1,
the frame image conversion unit converts the colors of pixels included in the acquired frame image so that a color gamut used in the converted frame image is wider than a color gamut used in the acquired frame image.
Image processing device. 2. The image processing device according to claim 1,
the frame image conversion unit converts the colors of pixels included in the acquired frame images into color tones that correspond to the color vision characteristics of the viewer.
Image processing device. 2. The image processing device according to claim 1,
the frame image conversion unit performs the conversion using a machine learning model selected based on given conditions from a plurality of machine learning models prepared in advance;
Image processing device. 7. The image processing device according to claim 6,
the frame image conversion unit performs the conversion using a machine learning model selected according to the type of application program that has drawn the acquired frame image.
Image processing device. 2. The image processing device according to claim 1,
The machine learning model is a model obtained by machine learning using two frame images obtained by drawing the same content under different drawing conditions, one of which is used as input data and the other as training data.
Image processing device. acquiring a frame image constituting a video to be presented to a user each time the frame image is drawn;
a step of converting the acquired frame image using a previously prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as a frame image to be presented to the user;
An image processing method comprising: acquiring a frame image constituting a video to be presented to a user each time the frame image is drawn;
a step of converting the acquired frame image using a previously prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as a frame image to be presented to the user;
A program that causes a computer to execute the following.