TWI884776B - Image data collection system, image model training method, and device for improving image resolution - Google Patents

Abstract

The embodiments of this application provide an image data collection system, an image model training method, and a device for improving image resolution. In this application, an image capturing device is used to capture an image of an object at different focal lengths to obtain a first image and a second image respectively, and the first image and the second image are processed to obtain a first processed image with high resolution and a second processed image with low resolution. Image alignment is performed on these processed images to obtain a high-resolution and low-resolution image pair. Many high-resolution and low-resolution image pairs are collected as a training image data set to train a model for upgrading low-resolution images to high-resolution images. The trained model can significantly improve the ability to restore image details.

Description

Image data acquisition system, image model training method, and device for improving image resolution

This application embodiment relates to image processing technology, and in particular to an image data acquisition system, an image model training method, and a device for improving image resolution.

High-resolution (HR) videos can bring better visual effects to the audience. With the development of optical imaging technology, optical couplers and high-speed communication technology, the capture, storage and projection of 8K videos have become mature technologies. However, high-resolution videos require larger storage space and transmission bandwidth, and the price of related equipment is also relatively expensive. Digital super-resolution (SR) technology is a very popular image processing technology today. The spirit of this technology is to use the spatial domain information or spatial frequency domain information of low-resolution (LR) images for upsampling to estimate the optical transfer function of the optical system, as shown in Figure 1. This is different from the traditional upsampling method based on interpolation (such as the nearest neighbor method, bilinear method, bicubic method, etc.).

Generally speaking, there are two approaches to super-resolution technology, namely optical methods and deep neural network (DNN) methods. Optical methods rely on understanding the optical system to improve resolution, while DNN methods use machine learning to learn some patterns from data and can adapt to a wider range of situations. Although optical methods can achieve higher interpretability, noise may cause errors in the deconvolution process. DNN methods can use more complex patterns and can achieve better performance than optical methods for natural images taken with cameras.

Super-resolution technology has emerged in recent years, most of which are based on deep learning methods. Traditional interpolation sampling methods usually only involve 4 to 9 pixels around the target pixel for interpolation operations, while deep learning networks constructed by many convolutional layers can analyze features in images, which means that they have a larger receptive field and therefore have better nonlinear mapping capabilities than traditional interpolation methods, which can achieve super-resolution goals. A large number of studies in recent years have also proved the effectiveness of deep learning methods.

High-resolution images provide more details, but their high pixel density increases the transmission bandwidth, video storage costs, and the cost of related products. Although the use of HR image sensors is the most direct way to obtain HR images, the manufacturing process and cost limitations of such sensors and optical equipment usually make this approach impractical in many occasions or large-scale deployments. With the continuous development of the application and accuracy requirements of imaging technology (such as image analysis, image display, microscopes, etc.), the demand for higher image resolution continues to increase. In addition, with the widespread development and application of super-resolution technology in video and image processing, it is crucial to develop super-resolution solutions that meet the temporal and spatial continuity of video content.

Today, most super-resolution datasets consist of "synthesized" data, which are low-resolution images generated by numerical methods such as bilinear and bicubic, as shown in Figure 2. Such datasets are easy to construct, but the downsampling process in real optical systems is more complex than these simple models. When operating on real images, DNNs trained on these synthetic datasets are generally unable to super-resolve high-frequency details to the same clarity and sharpness as synthetic LR images. Therefore, the use of synthetic training images has this weakness in producing high-resolution images.

This application embodiment provides an image data acquisition system, an image model training method, and a device for improving image resolution, which can enhance the ability to restore image details. The technical solution is as follows:

According to one aspect of an embodiment of the present application, an image data acquisition system is provided, comprising: an image capture device, used to capture an image of an object at a first focal length to obtain a first image, and to capture an image of the object at a second focal length to obtain a second image, wherein the first image and the second image have the same resolution; a storage device, used to store the first image and the second image captured by the image capture device; and a processing module, which obtains the first image from the storage device. image and the second image, for processing the first image to obtain a first processed image with a first resolution, and processing the second image to obtain a second processed image with a second resolution, wherein the first resolution is greater than the second resolution; and a registration module, for obtaining the first processed image and the second processed image from the processing module, and performing image alignment on the first processed image and the second processed image to obtain a high-resolution and low-resolution image pair.

According to another aspect of the embodiment of the present application, a method for training an image model is provided, comprising: obtaining an image data set through an optical system, the image data set comprising a plurality of high-resolution and low-resolution image pairs, each high-resolution and low-resolution image pair comprising a first training image having a first resolution obtained based on a first focal length and a second training image having a second resolution obtained based on a second focal length, the first training image and the second training image having the same or corresponding image content, and the first resolution is greater than the second resolution; and inputting the image data set into a neural network model to train a trained image model, wherein the first training image is used as an input of the neural network model, and the second training image is used as a training label.

According to another aspect of the embodiment of the present application, a device for improving image resolution is provided, comprising: an input unit for receiving a low-resolution image; a controller coupled to the input unit, wherein an image conversion model is configured in the controller, and the image conversion model is used to convert the low-resolution image into a high-resolution image, wherein the image conversion model is trained using an image data set, and the image data set includes a plurality of high-resolution and low-resolution image pairs (image pairs), each of the high-resolution and low-resolution image pairs includes a first training image (training image) with a first resolution obtained based on a first focal length. image) and a second training image with a second resolution obtained based on a second focal length, the first training image and the second training image have the same or corresponding image content, and the first resolution is greater than the second resolution; and an output unit coupled to the controller for outputting the high-resolution image, wherein the resolution of the high-resolution image is higher than the resolution of the low-resolution image.

The technical solution provided by this application embodiment may include the following beneficial effects:

In the embodiment of the present application, an image capture device is used to capture images of an object at different focal lengths to obtain a first image and a second image respectively, and the first image and the second image are processed to obtain a high-resolution first processed image and a low-resolution second processed image, and these processed images are aligned to obtain a high-resolution and low-resolution image pair. Many high-resolution and low-resolution image pairs are collected as a training image data set to train a model (such as a neural network model) for upgrading low-resolution images to high-resolution images. The trained model can significantly improve the ability to restore image details.

It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and do not limit the present application.

1: Object

2: Curtain

10: Image capture device

20: Storage

30: Computing equipment

32: Processing module

34:Register module

35: Cutting module

36: Standard Deviation Filter

82: Cosine pattern

83: Middle area

85, 85’: Spectrum center

86a, 86b: Peak

87a, 87b: Peak

100: Image data acquisition system

1901:UHD8k dataset

1902: Pre-training model

1903:DSD data set

1904: Dataset added from the movie

1905: VSR model

2400: Device for improving image resolution

2410: Input unit

2420: Controller

2422: Image conversion model

2430: Output unit

2500: High-resolution conversion system

Im1: First Image

Im2: Second Image

R1: First area

R2: Second area

S701~S708: Steps

S1201~S1208: Steps

In order to more clearly explain the technical solutions in the embodiments of this application, the following will briefly introduce the figures required for the description of the embodiments. Obviously, the figures described below are only some embodiments of this application. For those with ordinary knowledge in the relevant technical field, other figures can be obtained based on these figures without creative labor.

[Figure 1] Schematic diagram showing the principle of optical transfer function in super-resolution technology.

[Figure 2] Schematic diagram showing the generation process of synthetic training images.

[Figure 3] Schematic diagram showing the generation process of real training images.

[Figure 4] is a schematic diagram showing an image of an object captured by an image capture device according to an embodiment of the present application.

[Figure 5] shows a block diagram of an image data acquisition system according to an embodiment of the present application.

[Figure 6] is a schematic diagram showing the object image processing process according to the embodiment of the present application.

[Figure 7] shows a schematic diagram of the process of obtaining a pair of high-resolution and low-resolution images according to an embodiment of the present application.

[Figure 8] Schematic diagram showing the cosine pattern and the spectrum of the cosine pattern during the DSD amplification correction process.

[Figure 9] shows the process of image calibration in the Design Scene Dataset (DSD).

[Figure 10] shows an example of image calibration results in the Design Scene Dataset (DSD).

[Figure 11] shows a schematic diagram of obtaining the optimal cropping size based on the model training method.

[Figure 12] shows a schematic diagram of the cropping process to generate the synthetic dataset.

[Figure 13] shows the test results of training three DNN models using datasets with different cropping sizes.

[Figure 14] shows the test results of training the MANtiny model using datasets with different cropping sizes.

[Figure 15] shows the test results of training the RFDN model using datasets with different cropping sizes.

[Figure 16] shows the test results of training the SRCNN model using datasets with different cropping sizes.

[Figure 17] shows a schematic diagram of the process of obtaining the optimal cutting size based on the spectrum analysis method.

[Figure 18] A graph showing the correlation between spatial frequency and spectrum.

[Figure 19] shows a training flow chart of a video super-resolution model.

[Figure 20] shows a redundant image recognition flow chart.

[Figure 21] shows the gradient histogram of the image.

[Figure 22] shows a cross-section of the spectrum along the horizontal axis.

[Figure 23] shows the modulation transfer function (MTF) results obtained by various methods.

[Figure 24] shows a block diagram of a device for improving image resolution according to an embodiment of the present application.

[Figure 25] shows a schematic diagram of the configuration of a high-resolution conversion system according to an embodiment of the present application.

The following will combine the figures in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by people with ordinary knowledge in the relevant technical field without creative labor are within the scope of protection of this application.

In super-resolution (SR) technology, a model trained with synthetic training images usually cannot produce high-resolution (HR) images with the same clarity and sharpness as synthetic low-resolution (LR) images. In the embodiment of the present application, in order to allow the model to learn the transfer function in a real optical system, a "real" data set is established, as shown in Figure 3, and the model is trained with the real data set.

Please refer to Figures 4 to 6, Figure 4 shows a schematic diagram of using an image capture device to capture an image of an object according to an embodiment of the present application, Figure 5 shows a block diagram of an image data acquisition system according to an embodiment of the present application, and Figure 6 shows a schematic diagram of the object image processing process according to an embodiment of the present application. As shown in Figure 5, the image data acquisition system 100 of the embodiment of the present application includes an image capture device (e.g., a camera lens) 10, a storage 20 and a computing device 30, and the storage 20 can be configured in the image capture device 10, and can also be configured in the computing device 30. The computing device (e.g., computer) 30 is equipped with a processing module 32 and a registration module 34. The processing module 32 includes a cutting module 35. These modules 32, 34, and 35 can be implemented in hardware, software, firmware, or a combination of hardware and software.

As shown in FIG4 , the image capture device 10 is used to capture an image 1 of an object to obtain an image of the object 1. When performing image capture, the object 1 can be placed in front of a solid-color curtain 2 so that the captured image of the object 1 has a solid-color background to facilitate subsequent image processing. The image capture device 10 includes or is a zoom lens, which captures the image of the object 1 at a first focal length (e.g., focal length = 86 mm) to obtain a first image Im1, and captures the image of the object 1 at a second focal length (e.g., focal length = 43 mm) to obtain a second image Im2. The first image Im1 and the second image Im2 captured by the image capture device 10 have the same image size and the same resolution. When the first focal length of the image capture device 10 is greater than the second focal length, the size of the object 1 in the first image Im1 will be greater than the size of the object 1 in the second image Im2, as shown in FIG. 4.

The first image Im1 and the second image Im2 captured by the image capture device 10 can be stored in the memory 20 of the image capture device 10, or can be transmitted to the computing device 30 and stored in the memory 20 of the computing device 30. The memory 20 can be a non-volatile memory or a volatile memory.

The computing device 30 obtains the first image Im1 and the second image Im2 from the memory 20, and the processing module 32 of the computing device 30 is used to process the first image Im1 to obtain a first processed image with a first resolution, and to process the second image Im2 to obtain a second processed image with a second resolution.

The first resolution of the first processed image is greater than the second resolution of the second processed image. The image captured by the image capture device 10 at a longer focal length has richer local details, while the image captured at a shorter focal length covers a larger field of view, but the details are less clear. Therefore, it is desired to obtain a high-resolution image at a longer focal length and a low-resolution image at a shorter focal length. If the first focal length is f1 and the second focal length is f2, f1 and f2 have the following relationship: f1=A * f2, where A>1. Then, since it is desired to capture a high-resolution image at a long focal length and a low-resolution image at a short focal length, if the first resolution of the first processed image is X, then the second resolution of the second processed image can be X/A. That is, the magnification relationship between the first focal length and the second focal length can be used to determine the magnification relationship between the first resolution and the second resolution, because the magnification of the image is proportional to the focal length, and in this case, the image resolution is related to the image magnification. For example, the first resolution is twice the second resolution. For example, the first resolution is 8K and the second resolution is 4K. In other embodiments, the first resolution and the second resolution can also be other magnification relationships, not limited to a two-fold relationship. If A=2, the obtained first processed image and second processed image can be used to train a model suitable for improving the image resolution by two times.

Specifically, please refer to FIG6 , the cropping module 35 in the processing module 32 can crop the first area R1 in the first image Im1 to obtain the first processed image, and crop the second area R2 in the second image Im2 to obtain the second processed image, wherein the image content of the first area R1 corresponds to the image content of the second area R2, and the first area R1 is larger than the second area R2. Since the size of the first area R1 is larger than the size of the second area R2, the resolution of the first area R1 is greater than the resolution of the second area R2. If the first focal length is f1 and the second focal length is f2, f1 and f2 have the following relationship: f1=A * f2, where A>1. Then, if the resolution of the first area R1 is X, the resolution of the second area R2 can be X/A. If A=2, the obtained images of the first region R1 and the second region R2 can be used to train a model suitable for improving the image resolution by two times. For example, as shown in Figure 6, the resolution of the first region R1 is 1500×1500, and the resolution of the second region R2 is 750×750. That is, the first resolution of the first processed image corresponding to the first region R1 is greater than the second resolution of the second processed image corresponding to the second region R2.

The image data acquisition system 100 may further include a standard deviation filter 36 for determining whether to remove or retain the cropped image of the first region R1 based on the standard deviation of grayscale values of the first image Im1 and the standard deviation of grayscale values of the image of the first region R1. For example, if the standard deviation of grayscale values of the cropped image of the first region R1 is greater than (or equal to) R times (R is 0-1, such as 0.5) the standard deviation of grayscale values of the first image Im1, it means that the cropped image of the first region R1 contains the details of the first image Im1 and is therefore suitable as a training image for training the model. If the grayscale standard deviation of the cropped first region R1 is less than R times the grayscale standard deviation of the first image Im1 (R is 0~1, such as 0.5), it means that the cropped first region R1 lacks the details of the first image Im1, and the cropped area may be a background image or the like that is not suitable as a training image for training the model. Therefore, by using the standard deviation filter 36, suitable cropped images can be retained and unsuitable cropped images can be removed, so the quality of the training image dataset can be further improved. If the standard deviation filter 36 determines to retain the cropped first region R1, the region corresponding to the first region R1 of the first image Im1 in the second image Im2 will be further cropped, that is, the second region R2.

The registration module 34 obtains the first processed image and the second processed image from the processing module 32, and performs image alignment on the first processed image and the second processed image to obtain a high-resolution and low-resolution image pair. An appropriate algorithm may be used to align the first processed image and the second processed image, for example, the image alignment may be performed based on the difference between the phase map of spectrum of the first processed image and the second processed image. The registration module 34 can register the aligned first processed image and the second processed image as an image pair. This image pair is composed of a high-resolution image and a low-resolution image, so it is called a high-resolution and low-resolution image pair. This image pair can be stored in the memory 20 or other memory, and can be used as a training image pair for training a model (such as a neural network model). The second processed image is used as the input of the model, and the first processed image is used as the output of the model. The data set composed of many high-resolution and low-resolution image pairs can be called a designed scene dataset (DSD). Introducing DSD into the model training process is a valuable enhancement that can significantly improve the model's ability to recover image details when upscaling from low-resolution images to high-resolution images (e.g., 4k images to 8k images).

In the embodiment of the present application, an image capture device is used to capture images of an object at different focal lengths to obtain a first image and a second image respectively, and the first image and the second image are processed to obtain a high-resolution first processed image and a low-resolution second processed image, and these processed images are aligned to obtain a high-resolution and low-resolution image pair. Many high-resolution and low-resolution image pairs are collected as a training image data set to train a model (such as a neural network model) for upgrading low-resolution images to high-resolution images. The trained model can significantly improve the ability to restore image details.

FIG7 shows a schematic diagram of the process of obtaining a high-resolution and low-resolution image pair according to an embodiment of the present application. As shown in FIG7 , the process includes magnification calibration (step S701), obtaining a high-resolution image (step S702), changing the focal length (step S703), obtaining a low-resolution image (step S704), a cropping process (step S705), image registration (step S706), similarity determination (step S707), and storing image pairs (step S708).

Natural images captured by a camera have uncertainties, such as noise, distortion, aberration, and error caused by the environment. In order to reduce these uncertainties, the acquisition of data sets must be controlled in a designed environment. High-resolution images and low-resolution images are captured by changing the focal length of the zoom lens. For example, after capturing a high-resolution image (step S702), the focal length of the zoom lens can be changed (step S703) to shorten the focal length to twice the original to obtain a low-resolution image (step S704). Afterwards, the obtained high-resolution image and low-resolution image are subjected to a cropping process (step S705). During the cropping process, the image sizes of the high-resolution image and the low-resolution image maintain a certain proportional relationship. For example, when A=2, if the high-resolution image is 1500×1500, the low-resolution image is 750×750; if the high-resolution image is 400×400, the low-resolution image is 200×200.

Among them, the curtain 2 can be set to a white background, which has two advantages. First, the grayscale range of the white background is smaller, which is convenient for using the above-mentioned standard deviation filter 36 to determine whether the cropped area image contains image details. Second, the depth of focus is smaller in the case of high resolution, which means that the background becomes more blurred. If a more blurred HR is put into the training data set, the model will learn how to blur the image, which is not the goal of training the model. The white background exhibits similar characteristics when it is focused and defocused. It can reduce the error caused by defocus. In order to reduce the impact of distortion, only the middle area of the image can be selected for cropping.

The preparation flow chart of the design scene dataset (DSD) is shown in Figure 7. It contains two core steps: magnification correction (step S701) and image registration (step S706). Through these steps S701 and S706, the uncertainty caused by zoom lens adjustment and camera shift can be reduced.

Among them, in the magnification calibration (step S701), the magnification calibration can be performed on the calibration image to determine the position of the focus knob of the image capture device at the first focal length and the second focal length, so that the first image Im1 and the second image Im2 can be captured at the first focal length and the second focal length, respectively. The calibration of the magnification can be performed by two methods: cosine pattern spectrum and Fourier Mellin transform. Based on this, a focal length restriction mechanism is added so that the focus knob can be turned to the same position each time the image is magnified. Preferably, for example, the deviation between the calibration result and the double magnification is less than 0.0025. As shown in Figure 8, the calibration image (e.g., cosine pattern 82) is used to calibrate the magnification. A middle region 83 is cropped from the cosine pattern 82 in the HR image and the LR image, respectively. The size of the middle region 83 of the HR image is the same as that of the middle region 83 of the LR image. The middle region 83 of the HR image and the LR image are subjected to fast Fourier transform (FFT) to obtain their spectrum diagrams, respectively, as shown in the two diagrams at the bottom of FIG8 . For the HR spectrum, the distances between the two peaks 86a and 86b and the spectrum center 85 are equal. Similarly, for the LR spectrum, the distances between the two peaks 87a and 87b and the spectrum center 85' are equal. In the spectrum of the high-resolution (HR) image, the larger the period and the lower the frequency, the closer the positions of the two peaks are to the spectrum center. By using the properties of Fourier transform, the magnification can be calculated by the peak distance in the spectrum of the high-resolution image and the low-resolution image. For example, if the magnification is two times (i.e., A=2), the distance (or average distance) between the two peaks 87a, 87b in the LR spectrum and the spectrum center 85' will be twice the distance (or average distance) between the two peaks 86a, 86b in the HR spectrum and its spectrum center 85.

In DSD, the present application obtains high-resolution images and low-resolution images by changing the focal length of the camera, but the field of view (FOV) of the camera may change, and directly cropping the image pair may cause the data set to be contaminated. In image registration (step S706), the two cropped images are placed in the same field of view, and the image alignment can be performed based on the difference between the spectral phase maps of the two cropped images, as shown in FIG9 . That is, in step S707, if the error between the two cropped images in this field of view is too large (the difference between the two exceeds a certain value, such as 5%), then re-crop or reposition; if the error between the two cropped images is small (the similarity (correlation) between the two is greater than a certain value, such as 95%), then they can be stored as an image pair (step S708). Figure 10 shows an example of this image calibration result.

Since computers may not have the computing power required for high-resolution image datasets (e.g., datasets consisting of 4k to 8k images), in order to make the dataset trainable, the images can be cropped to a smaller size. In order to determine which image size can produce satisfactory training performance, two strategies for determining the optimal image size are proposed below.

Model training method:

Obtaining the minimum or optimal cropped size that does not affect the performance of the model can be achieved through the following steps: obtaining a high-resolution image and a low-resolution image with image content corresponding to the high-resolution image; cropping the high-resolution image at multiple different sizes and cropping the same area in the low-resolution image to obtain high-resolution and low-resolution image pairs; determining the cropping size based on the training results of the model for the high-resolution and low-resolution image pairs at each size.

Specifically, this method uses training datasets with different cropping sizes to train the model. For the training dataset, images can be collected from an existing image dataset (UHD8k dataset), which provides 2029 8K (7680x4320) images. Although the images in the dataset are of high resolution, they need to be cropped into smaller sizes due to the limitations of computer hardware and training speed. For example, the dataset is cropped into different square sizes: 2000, 1000, 800, 400, 200, 100, 80, 70, 60, 50, 30, 20. In order for the features in each cropping size to have the same properties, the small-size dataset is cropped from the large-size dataset. To ensure that the relationship between cropping size and training performance is applicable to different CNN models, three different models, MANtiny, SRCNN, and RFDN, are selected for testing. The model training method flow is shown in Figure 11.

Taking 8K images as an example, the cropping process is as shown in Figure 12. As shown in Figure 12, starting from the 8K image, the 8K image is randomly cropped (step S1201). Continue this process until the image of the minimum size is stored. In steps S1202, S1203 and S1204, a standard deviation test is performed. The standard deviation test is to test whether the image has some features that allow the model to learn. Generally speaking, the background is a color block with a small grayscale change. In order to ensure that the features in the 8k image are cropped, the grayscale value standard deviation can be used as an indicator. First, the grayscale value standard deviation Std _cr of the cropped image is calculated (step S1202), and the grayscale value standard deviation Std _8k of the 8k image is calculated (step S1203). A comparison is performed between the two in step S1204. For example, if the grayscale value standard deviation of the cropped image is greater than (or equal to) R times the grayscale value standard deviation of the 8K image (R is 0~1, such as 0.5), it means that the cropped image contains the details of the 8K image and is suitable as a training image for training the model. If the grayscale value standard deviation of the cropped image is less than R times the grayscale value standard deviation of the 8K image (R is 0~1, such as 0.5), it means that the cropped image lacks the details of the 8K image, and the cropped area may be a background image, etc., which is not suitable as a training image for training the model, and it is necessary to return to step S1201 for re-cropping. In addition, the 8k image is downsampled (step S1205) to obtain a 4k image (step S1206). For example, the Lanczos method can be applied to generate a low-resolution (e.g., 0.5x resolution) image. Then, a region having the same or similar image content as the cropped image obtained from step S1201 and satisfying the condition of step S1204 is cropped from the 4k image (step S1207), thereby obtaining a cropped data pair (step S1208).

For each model, the fluctuation is less than 1% after cropping to size = 100x100, and the results are shown in Figures 13 to 16.

Spectrum analysis method:

Obtaining the minimum or optimal cropping size that does not affect the model performance can be achieved through the following steps: obtaining a high-resolution image and a low-resolution image having image content corresponding to the high-resolution image; performing fast Fourier transformation on the high-resolution image and the low-resolution image to obtain a high-resolution image spectrum and a low-resolution image spectrum respectively; cropping the high-resolution image and the low-resolution image at a plurality of different sizes for the low-frequency region in the high-resolution image and the low-resolution image; and determining the cropping size from these sizes based on the correlation between the high-resolution image spectrum and the low-resolution image spectrum at these sizes.

Specifically, an image is composed of many plane waves of different frequencies. The process of image cropping can be viewed as blocking the signal outside the cropped region. The larger the crop size, the lower frequency structure can be included. We want to understand at what frequency the high-resolution image and the low-resolution image begin to differ in the spectrum. To understand this, the low-frequency region is cropped into different sizes. The entire flow chart is shown in Figure 17. By calculating the spectral correlation of multiple images in an existing dataset (e.g., UHD8k dataset), the relationship between the correlation and frequency can be plotted, as shown in Figure 18. After determining the frequency, we can analyze how the cropping process affects the spectrum. When cropping, the spectrum is distorted as the size becomes smaller, resulting in information loss. By cropping the analog signal at different sizes, it is possible to determine which crop size is safe for preventing information loss. As shown in the results in Figure 18, the difference between the 4k and 8k spectrum becomes significant at frequencies > 0.052 1/pixel.

From the above two strategies, it can be concluded that the dataset is sufficient when the crop size > 100x100. However, to ensure stability in special cases, crop size = 400x400 can be chosen as the standard.

The present application embodiment provides a method for training an image model, comprising: obtaining an image data set through an optical system, the image data set comprising a plurality of high-resolution and low-resolution image pairs, each high-resolution and low-resolution image pair comprising a first training image (training image) having a first resolution obtained based on a first focal length and a second training image having a second resolution obtained based on a second focal length, the first training image and the second training image having the same or corresponding image content, and the first resolution is greater than the second resolution; and inputting the image data set into a neural network model to train a trained image model, wherein the first training image is used as the input of the neural network model, and the second training image is used as a training label.

FIG19 shows a training flow chart of a video super-resolution model. First, the UHD8k dataset 1901 is used for training. The UHD8k dataset is a synthetic dataset. The synthetic dataset can be used for pre-training to obtain a pre-trained model 1902. Then, the pre-trained model 1902 is trained using a designed scene dataset (i.e., DSD) 1903 composed of real data to obtain the final video super-resolution (VSR) model 1905. In addition to using DSD, more high-resolution and low-resolution image pairs can be obtained from the video as a dataset (this belongs to the synthetic dataset) 1904 for model training to obtain the VSR model. The video super-resolution model can be used to upgrade the resolution of a video/movie, where each frame is passed through the super-resolution model, increasing the resolution by, for example, twice.

In the process of obtaining more high-resolution and low-resolution image pairs from the film as data sets to participate in model training, representative images are mainly extracted from the film to generate high-resolution and low-resolution versions of the representative images to participate in the training. In order to prevent repeated images from causing over-fitting of the model training results, at least one of the redundant images and blurred images in the film can be further excluded, and the remaining images in the film can be used as representative images. Specifically, redundant images can be identified by comparing the similarity between adjacent image frames, and blurred images can be identified by evaluating the clarity of the image.

Before extracting images from a video, the input high-resolution (e.g., 8K) video can be downsampled (e.g., reduced to 1/16 of the original) using the bicubic method to reduce the computation time of subsequent steps. After identifying redundant and blurred images in the video, the frame numbers of these unqualified frames are recorded to exclude these frames as data sets for model training.

The redundant image identification process is shown in Figure 20. A sliding window containing three frames moves on each frame of the video. The structural similarity index (SSIM) between the first frame and the second frame, and the SSIM between the third frame and the second frame are calculated. If these SSIM values are both higher than the redundancy threshold ( _Tr ), the second frame is identified as a redundant image.

That is, if SSIM(f ₂ ,f ₁ )> _Tr and SSIM(f ₂ ,f ₃ )> _Tr , then f ₂ is a redundant image.

In subsequent calculations, if the second frame is identified as a redundant image, the first frame is retained. Otherwise, the second frame will become the new first frame.

In blur detection, first set the threshold (S _intensity ) and use the average grayscale value of the frame to detect fade-in and fade-out frames. If the SSIM between two adjacent frames is less than the threshold (T _sc ), it means that a scene change is detected. That is, if SSIM ( _fi , fi _{- 1} ) < T _sc , _fi is taken as the first frame of the new scene. Then, calculate the clarity of the first frame in the scene and use it as the reference clarity (S _ref1 , S _ref2 ). If the clarity of the next frame (S _i1 , S _i2 ) is less than the product of the blur threshold (T _bl ) and the reference clarity, the frame is called blurred. That is, if _Si1 < _Tbl × _Sref1 or _Si2 < _Tbl × _Sref2 , then _fi is blurred. The sharpness of an image can be calculated by two methods: a gradient-based method and a spectrum-based method. The reference sharpness _Sref1 and _Sref2 can be obtained by the two methods, respectively. _Si1 and _Si2 are also obtained by the two methods, respectively.

The following introduces the gradient-based method. If an image contains a large number of sharp edges, it can be considered that the image is a sharp image, which means that the intensity of the gradient map of the sharp image will be higher than the intensity of the gradient map of the blurred image. The gradient map of the image can be obtained through Laplacian derivatives, and the gradient intensity distribution can be represented by a histogram. As shown in Figure 21, (a) on the left side of Figure 21 is the gradient histogram of the blurred image, and (b) on the right side of Figure 21 is the gradient histogram of the sharp image. For the distribution of gradients, clear images have a larger variance than blurred images, which means that the standard deviation can also be used as an indicator of sharpness. In the gradient-based method, the following two indicators can be used to calculate the clarity of the image:

Intensity (S _Ig ): The gradients in the gradient histogram are sorted by intensity, the top 0.1% are selected, and then their average is calculated as the clear intensity.

Variance ( _SV ): standard deviation of the gradient histogram.

The following introduces the spectrum-based method. Fourier transform can display the spectrum of an image. Figure 22 shows a spectrum cross-section along the horizontal axis (such as the x-axis). (a) on the left side of Figure 22 is a spectrum cross-section of a clear image, and (b) on the right side of Figure 22 is a spectrum cross-section of a blurred image. The cutoff frequency of a clear image is larger than that of a blurred image because high-frequency signals are suppressed in a blurred image. In addition, the bandwidth of the DC region of a clear image is smaller than that of a blurred image. The response of the low-frequency region of a clear image is weaker than that of a blurred image because the intensity diffuses to the high-frequency region. In the spectrum-based method, the following two indicators can be used to calculate the clarity of an image:

Coefficient of variation (COV): COV can be calculated as follows: σ _x and σ _y are the standard deviations of the intensity on the x-axis and y-axis of the spectrum graph, respectively. _{μ x} and μ _y are the mean values of the intensity on the x-axis and y-axis, respectively.

Intensity: Normalize the spectrum intensity to the range of 0~1, and then calculate the sum of the spectrum using the following formula. Collect the weak responses on the x-axis and y-axis of the spectrum graph, i.e. sum(I _x <0.0001) and sum(I _y <0.0001), respectively. w and h are the width and height of the spectrum. The intensity (S _Is ) of the blurred image is higher because its response to high frequencies is weaker than that of the clear image.

The USAF resolution benchmark is used here to analyze the improvement of the proposed method in terms of resolution capability. From the values of SSIM, PSNR, and DoM, it can be seen that the MANtiny model trained with DSD outperforms the model trained with data generated by the Lanczos3 interpolation method, as shown in Table 1 below.

The modulation transfer function (MTF) of each method is shown in Figure 23. The x-axis is the spatial frequency, which can be regarded as the density of the pattern. The y-axis is the contrast, which represents the ability of the system to resolve dense patterns. When lp/mm<2.24, the method proposed in this application (i.e., DSD) has the same performance as 8k, which means that the method of this application enhances the resolution at this stage. The model trained with lanczos3 is no different from 4k and interpolation methods because it learns how to solve the degradation caused by lanczso3 downsampling, rather than the degradation process in the optical system.

As shown in FIG. 24 , the present application embodiment provides a device 2400 for improving image resolution, which includes: an input unit 2410, a controller 2420, and an output unit 2430. The input unit 2410 is used to receive a low-resolution image. The input unit 2410 includes but is not limited to a wired or wireless input interface. The wired input interface includes a USB-C transmission interface, etc., and the wireless input interface includes a WI-FI, Bluetooth, cellular network transmission interface, etc. The controller 2420 is coupled to the input unit 2402, and the controller can also be a controller with arithmetic processing logic (for example, a central processing unit (CPU) or a graphics processing unit (GPU)). An image conversion model 2422 is configured in the controller 240, and the image conversion model 2422 is used to convert the low-resolution image into a high-resolution image, wherein the image conversion model 2422 is trained using an image data set, and the image data set includes a plurality of high-resolution and low-resolution image pairs, each of which includes a first training image with a first resolution obtained based on a first focal length and a second training image with a second resolution obtained based on a second focal length, and the first training image and the second training image have the same or corresponding image content, and the first resolution is greater than the second resolution. The output unit 2430 is coupled to the controller 2420, and the output unit 2430 includes but is not limited to a display interface, and can also be an output interface coupled to a storage unit. The output unit 2430 is used to output the high-resolution image, wherein the resolution of the high-resolution image is higher than the resolution of the low-resolution image.

FIG25 shows a schematic diagram of the configuration of a high-resolution conversion system 2500 according to an embodiment of the present application. Using the high-resolution conversion model trained by the embodiment of the present application, the input low-resolution image or video can be upgraded to a high-resolution image or video for output. For example, a 4K image can be upgraded to a high-quality image with a resolution of 8K. Furthermore, the designed scene data set (i.e., DSD) obtained by applying the concept of the present invention can be input into the trained high-resolution conversion model (such as the transfer function model shown in FIG22 ) in the high-resolution conversion system 2500 at any time to further optimize the high-resolution conversion model, improve the quality of the image or video outputted by it, and enhance the flexibility of the high-resolution conversion model in training. In addition, the images or videos converted by the high-resolution conversion model can be directly projected or displayed in real time, so that users can instantly see the effect of the image conversion, reducing the time users have to wait for the conversion process.

Although the present disclosure has been disclosed as above with the preferred embodiment, it is not intended to limit the present disclosure. A person with ordinary knowledge in the technical field to which the present disclosure belongs can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the scope defined by the attached patent application.

10: Image capture device

20: Storage

30: Computing equipment

32: Processing module

34:Register module

35: Cutting module

100: Image data acquisition system

Claims (20)

An image data acquisition system includes: an image capture device for capturing an image of an object at a first focal length to obtain a first image, and capturing an image of the object at a second focal length to obtain a second image, wherein the first image and the second image have the same resolution; a memory for storing the first image and the second image captured by the image capture device; a processing module for acquiring the first image and the second image from the memory, and processing the first image to obtain a first processed image with a first resolution, and processing the second image to obtain a second processed image with a second resolution, wherein the first resolution is greater than the second resolution; and A registration module obtains the first processed image and the second processed image from the processing module, and performs image alignment on the first processed image and the second processed image to obtain a high-resolution and low-resolution image pair, wherein the registration module performs image alignment based on the difference between the phase map of spectrum of the first processed image and the second processed image, wherein after the image alignment, if the correlation between the first processed image and the second processed image is greater than a certain value, the registration module stores the first processed image and the second processed image as the high-resolution and low-resolution image pair. An image data acquisition system as described in claim 1, wherein the processing module includes a cropping module, which is used to crop a first area in the first image to obtain the first processed image, and to crop a second area in the second image to obtain the second processed image, the image content of the first area corresponds to the image content of the second area, and the resolution of the first area is greater than the resolution of the second area. An image data acquisition system as described in claim 2, wherein the first focal length is f1, the second focal length is f2, then f1 = A * f2, wherein A > 1, and wherein the resolution of the first area is X, and the resolution of the second area is X/A. An image data acquisition system as described in claim 3, wherein A=2, and the obtained high-resolution and low-resolution image pairs are used to train a model suitable for increasing the image resolution by twice. An image data acquisition system as described in claim 2, wherein a resolution of the first area is greater than or equal to 100×100. The image data acquisition system as described in claim 2 further includes a standard deviation filter for determining whether to discard or retain the cropped image of the first area based on the standard deviation of the grayscale value of the first image and the standard deviation of the grayscale value of the image of the first area. An image data acquisition system as described in claim 1, wherein the first focal length and the second focal length of the image capture device are determined by performing magnification calibration on a calibration image. An image data acquisition system as described in claim 7, wherein the magnification correction of the correction image is implemented based on cosine pattern spectrum and Fourier Mellin transform. A method for training an image model, comprising: Obtaining an image dataset through an optical system, the image dataset comprising a plurality of high-resolution and low-resolution image pairs, each of which comprises a first training image having a first resolution based on a first focal length and a second training image having a second resolution based on a second focal length, the first training image and the second training image having the same or corresponding image content, and the first resolution is greater than the second resolution; Inputting the image dataset into a neural network model to train a trained image model, wherein the first training image is used as an input of the neural network model, and the second training image is used as a training label; and Image alignment is performed based on the difference between the phase map of spectrum of the first training image and the second training image, wherein after the image alignment, if the correlation between the first training image and the second training image is greater than a certain value, the first training image and the second training image are stored as the high-resolution and low-resolution image pair. The image model training method as described in claim 9 further includes: Cropping a first area in a first image to obtain the first training image; Cropping a second area in a second image to obtain the second training image, wherein the image content of the first area corresponds to the image content of the second area, and the resolution of the first area is greater than the resolution of the second area. The image model training method as described in claim 10, wherein the first focal length is f1, the second focal length is f2, then f1 = A * f2, wherein A > 1, and wherein the resolution of the first area is X, and the resolution of the second area is X/A. The image model training method as described in claim 11, wherein A=2, and the trained image model is a model suitable for improving the image resolution by two times. The image model training method as described in claim 10 further includes: Determining whether to remove or retain the cropped image of the first area based on the standard deviation of grayscale values of the first image and the standard deviation of grayscale values of the image of the first area. The image model training method as described in claim 9 further includes: determining the minimum cropped size of the image in the image dataset, which includes: Obtaining a high-resolution image and a low-resolution image having image content corresponding to the high-resolution image; Cropping the high-resolution image at multiple different sizes and capturing the same area in the low-resolution image to obtain high-resolution and low-resolution image pairs; Based on the training results of the model for the high-resolution and low-resolution image pairs at each size, determining the minimum cropped size. The training method of the image model as described in claim 9 further includes: determining the minimum cropping size of the image in the image dataset, which includes: Obtaining a high-resolution image and a low-resolution image having image content corresponding to the high-resolution image; Performing a fast Fourier transform on the high-resolution image and the low-resolution image to obtain a high-resolution image spectrum and a low-resolution image spectrum, respectively; For the low-frequency region in the high-resolution image and the low-resolution image, cropping the high-resolution image and the low-resolution image at a plurality of different sizes; Based on the correlation between the high-resolution image spectrum and the low-resolution image spectrum at these sizes, determining the minimum cropping size from these sizes. The image model training method as described in claim 9 further includes: Extracting a representative image from a video; and Generating a high-resolution version and a low-resolution version of the representative image to participate in the training of the neural network model. The training method of the image model as described in claim 16, wherein the step of extracting the representative image from the video includes: Identifying redundant images in the video and/or identifying blurred images in the video; and Excluding at least one of the redundant images and blurred images in the video, and using the remaining images in the video as the representative images. The method for training an image model as described in claim 17, wherein the step of identifying redundant images in the video includes: Identifying the redundant images by comparing the similarities between adjacent image frames. The image model training method as described in claim 17, wherein the step of identifying the blurred image in the video includes: Identifying the blurred image by evaluating the clarity of the image. A device for improving image resolution, comprising: an input unit for receiving a low-resolution image; a controller coupled to the input unit, wherein an image conversion model is arranged in the controller, and the image conversion model is used to convert the low-resolution image into a high-resolution image, wherein the image conversion model is obtained by training an image data set, and the image data set includes a plurality of high-resolution and low-resolution image pairs (image pairs), each of which includes a first training image (training image) with a first resolution obtained based on a first focal length. image) and a second training image with a second resolution obtained based on a second focal length, the first training image and the second training image have the same or corresponding image content, the first resolution is greater than the second resolution, wherein the first training image and the second training image are images obtained by image alignment and satisfying a similarity condition, the image alignment is performed based on the difference between the first training image and the second training image's phase map of spectrum, and after the image alignment, if the correlation between the first training image and the second training image is greater than a certain value, the first training image and the second training image are stored as the high-resolution and low-resolution image pair because they satisfy the similarity condition; and An output unit is coupled to the controller and is used to output the high-resolution image, wherein the resolution of the high-resolution image is higher than the resolution of the low-resolution image.