WO2022193132A1 - Image detection method and apparatus, and electronic device - Google Patents

Abstract

An image detection method and apparatus, and an electronic device. The image detection method comprises: acquiring an image to be subjected to detection; on the basis of an image feature of said image, predicting, by using a pre-trained parameter prediction model, a parameter of an image processing algorithm used for processing said image, so as to generate a parameter predicted value; adjusting the parameter of the image processing algorithm on the basis of the parameter predicted value; performing, by using the image processing algorithm that has been subjected to parameter adjustment, image processing on said image, so as to generate a processed image; and inputting the processed image into a pre-trained image detection model, so as to generate a detection result. By means of the method, the accuracy of a detection result of an image detection model can be improved.

Description

Image detection method, device and electronic device technical field

The embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to an image detection method, apparatus, and electronic device.

Background technique

With the development of science and technology, artificial intelligence (AI, Artificial Intelligence) technology has been improved by leaps and bounds. In some artificial intelligence technologies, machine learning methods are usually used to construct initial models of various structures, such as neural network models, support vector machine models, and decision tree models. Then, the initial model is trained by training samples to realize functions such as image detection, speech recognition, etc.

In the computer vision technology based on image detection, a perception model is obtained by training an initial model such as a neural network to realize image detection tasks such as scene recognition, object detection or image segmentation. Usually, the perceptual model extracts and detects the input image to output the detection result. Therefore, the perceptual model has specific requirements for the brightness, color, texture and other characteristics of the input image, such as clear image texture, moderate brightness, etc. . In scenes such as night, fog, glare or dust, the indicators used to form the brightness, color, texture and other characteristics of the image change, resulting in too dark image, unclear image texture, etc., resulting in detection results output by the perception model There is a large deviation from the real results. Therefore, for images collected in various scenes, how to enable the perception model to output relatively accurate detection results has become a problem to be solved.

SUMMARY OF THE INVENTION

The image detection method, device and electronic device provided by the present application can enable the image detection model to output relatively accurate detection results for images collected in various scenarios.

To achieve the above object, the application adopts the following technical solutions:

In a first aspect, an embodiment of the present application provides an image detection method, the image detection method includes: acquiring an image to be detected; Predict the parameters of the image processing algorithm of the image to be detected, and generate a parameter prediction value; based on the parameter prediction value, adjust the parameters of the image processing algorithm; use the image processing algorithm after parameter adjustment to perform image processing on the image to be detected , generating a processed image; inputting the processed image into a pre-trained image detection model to generate a detection result.

In this embodiment of the present application, a parameter prediction model is set to predict the parameters of the image processing algorithm, and the image processing algorithm uses the parameter prediction value to process the image to be detected, so that the dynamic range, noise, contrast and other parameters of the image input to the image detection model , which is the same or similar to the dynamic range, noise level, local contrast and other parameters of the sample image used to train the image detection model, which is beneficial to improve the accuracy of the detection result of the image detection model.

Based on the first aspect, in a possible implementation manner, the parameter prediction model is obtained by comparing the annotation information of the sample image with the detection result of the sample image by the image detection model, and training based on the comparison result.

Based on the first aspect, in a possible implementation manner, the parameter prediction model is obtained by training the following steps: comparing the detection result of the sample image with the annotation information of the sample image to obtain the comparison result; Iteratively adjusts the parameters of the parameter prediction model based on the comparison result; and saves the parameters of the parameter prediction model when a preset condition is satisfied.

The preset conditions here may include but are not limited to: the error is less than or equal to a preset threshold, or the number of iterations is greater than or equal to a preset threshold.

Based on the first aspect, in a possible implementation manner, the comparison result is an error, and the iteratively adjusting the parameters of the parameter prediction model based on the comparison result includes: based on the detection result of the sample image and the According to the error between the label information of the sample images, a target loss function is constructed, wherein the target loss function includes the parameters to be adjusted in the parameter prediction model; based on the target loss function, the back propagation algorithm and gradient A descent algorithm that iteratively adjusts the parameters of the image processing algorithm.

Based on the first aspect, in a possible implementation manner, the parameters of the image processing algorithm for processing the to-be-detected image are performed using a pre-trained parameter prediction model based on the image features of the to-be-detected image. Predicting, generating a parameter prediction value includes: performing feature extraction on the to-be-detected image to generate a feature tensor; inputting the feature tensor into the parameter prediction model to generate the parameter prediction value.

Based on the first aspect, in a possible implementation manner, the parameters of the image processing algorithm for processing the to-be-detected image are performed using a pre-trained parameter prediction model based on the image features of the to-be-detected image. Predicting, generating a parameter prediction value includes: inputting the to-be-detected image into the parameter prediction model to generate the parameter prediction value.

Based on the first aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm, and an image noise reduction algorithm.

Based on the first aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence levels, and prediction of motion trajectories of target objects .

In a second aspect, an embodiment of the present application provides an image detection device, the image detection device includes: an acquisition module configured to acquire an image to be detected; a generation module configured to use a pre- The trained parameter prediction model predicts the parameters of the image processing algorithm used to process the image to be detected, and generates parameter prediction values; an adjustment module is configured to adjust the parameters of the image processing algorithm based on the parameter prediction values. a processing module, configured to perform image processing on the image to be detected using an image processing algorithm after parameter adjustment, to generate a processed image; a detection module, configured to input the processed image to a pre-trained image Detect the model and generate detection results.

Based on the second aspect, in a possible implementation manner, the parameter prediction model is obtained by comparing the annotation information of the sample image with the detection result of the sample image by the image detection model, and training based on the comparison result.

Based on the second aspect, in a possible implementation manner, the image detection apparatus further includes a parameter prediction model training module, and the parameter prediction model training module includes: a comparison sub-module configured to detect the sample image The result is compared with the annotation information of the sample image to obtain the comparison result adjustment sub-module, which is configured to iteratively adjust the parameters of the parameter prediction model based on the comparison result; the saving sub-module is configured to be in a preset condition. When satisfied, the parameters of the parameter prediction model are saved.

Based on the second aspect, in a possible implementation manner, the comparison result is an error, and the adjustment sub-module is further configured to: based on the difference between the detection result of the sample image and the annotation information of the sample image For the error, a target loss function is constructed, wherein the target loss function includes the parameters to be adjusted in the parameter prediction model; based on the target loss function, the image processing is iteratively adjusted by using a back-propagation algorithm and a gradient descent algorithm parameters of the algorithm.

Based on the second aspect, in a possible implementation manner, the generating module is configured to: perform feature extraction on the to-be-detected image to generate a feature tensor; input the feature tensor into the parameter prediction model , to generate the predicted value of the parameter.

Based on the second aspect, in a possible implementation manner, the generating module is configured to: input the image to be detected into the parameter prediction model, and generate the parameter prediction value.

Based on the second aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm, and an image noise reduction algorithm.

Based on the second aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting a confidence level, and predicting a motion trajectory of the target object .

In a third aspect, an embodiment of the present application provides a method for training a parameter prediction model, the method comprising: based on an image feature of a sample image, using the parameter prediction model, adjusting the parameters of an image processing algorithm used to process the sample image performing prediction to generate a parameter prediction value; adjusting the parameters of the image processing algorithm based on the parameter prediction value; performing image processing on the sample image by using the parameter-adjusted image processing algorithm to generate a processed image; The processed image is input to a pre-trained image detection model to generate a detection result; the error between the detection result and the labeling information of the sample image is compared to obtain a comparison result; based on the comparison result, the parameter prediction is adjusted The parameters of the model; when the preset conditions are satisfied, the parameters of the parameter prediction model are saved.

Based on the third aspect, in a possible implementation manner, the comparison result is an error, and adjusting the parameters of the parameter prediction model based on the comparison result includes: based on the detection result and the sample image The error between the labeling information, constructs a target loss function, wherein the target loss function includes the parameters to be adjusted in the parameter prediction model; based on the target loss function, using the back-propagation algorithm and the gradient descent algorithm, Iteratively adjust the parameters of the parameter prediction model.

Based on the third aspect, in a possible implementation manner, the comparison result is an error, and adjusting the parameters of the parameter prediction model based on the comparison result includes: based on the detection result and the sample image The error between the labeling information, constructs a target loss function, wherein the target loss function includes the parameters to be adjusted in the parameter prediction model; based on the target loss function, using the back-propagation algorithm and the gradient descent algorithm, Iteratively adjust the parameters of the parameter prediction model.

Based on the third aspect, in a possible implementation manner, the parameter prediction model to be trained is used to predict the parameters of the image processing algorithm for processing the sample image based on the image features of the sample image, and the parameters are generated. The predicted value includes: performing feature extraction on the sample image to generate a feature tensor; inputting the feature tensor into the parameter prediction model to generate the parameter predicted value.

Based on the third aspect, in a possible implementation manner, the parameter prediction model to be trained is used to predict the parameters of the image processing algorithm for processing the sample image based on the image features of the sample image, and the parameters are generated. The predicted value includes: inputting the sample image into the parameter prediction model to generate the parameter predicted value.

Based on the third aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm, and an image noise reduction algorithm.

Based on the third aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting a confidence level, and predicting a motion trajectory of the target object .

In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device includes: a camera device for acquiring an image to be detected; a parameter prediction device for using pre-trained parameters based on image features of the image to be detected a prediction model, for predicting parameters of an image processing algorithm used to process the image to be detected, and generating a parameter prediction value; an image signal processor for adjusting the parameters of the image processing algorithm based on the parameter prediction value, using The parameter-adjusted image processing algorithm performs image processing on the to-be-detected image to generate a processed image; an artificial intelligence processor is used to input the processed image into a pre-trained image detection model to generate a detection result.

In a fifth aspect, an embodiment of the present application provides an image detection device, the image detection device includes one or more processors and a memory; the memory is coupled to the processor, and the memory is used to store one or more programs ; the one or more processors are configured to run the one or more programs to implement the method according to the first aspect.

In a sixth aspect, an embodiment of the present application provides an apparatus for training a parameter prediction model, the apparatus for training a parameter prediction model includes one or more processors and a memory; the memory is coupled to the processor, and the The memory is used to store one or more programs; the one or more processors are used to execute the one or more programs to implement the method according to the third aspect.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is used to implement the method according to the first aspect when the computer program is executed by at least one processor. .

In an eighth aspect, an embodiment of the present application provides a computer program product, which is used to implement the method according to the first aspect when the computer program product is executed by at least one processor.

It should be understood that the second to eighth aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding feasible implementation manner are similar, and will not be repeated here.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 2a is a schematic structural diagram of the parameter prediction apparatus shown in FIG. 1 provided by an embodiment of the present application;

FIG. 2b is another schematic structural diagram of the parameter prediction apparatus shown in FIG. 1 provided by an embodiment of the present application;

3 is a schematic structural diagram of a vehicle provided by an embodiment of the present application;

FIG. 4 is an interaction flowchart between components in each electronic device as shown in FIG. 1 provided by an embodiment of the present application;

5 is another schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 6 is an interaction flowchart between components in the electronic device as shown in FIG. 5 provided by an embodiment of the present application;

7 is a schematic diagram of a method for performing feature extraction on an image to be detected to generate a feature tensor provided by an embodiment of the present application;

8 is a schematic diagram of parameters included in an image processing algorithm provided by an embodiment of the present application;

9 is a schematic diagram of a system architecture including an electronic device for training an image prediction model provided by an embodiment of the present application;

10 is a flowchart of a method for training a parameter prediction model provided by an embodiment of the present application;

11 is a flowchart of an image detection method provided by an embodiment of the present application;

12 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

FIG. 13 is a schematic structural diagram of an apparatus for training a parameter prediction model provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the following description, reference is made to the accompanying drawings which form a part of this application and which illustrate, by way of illustration, specific aspects of embodiments of the application, or specific aspects of which embodiments of the application may be used. It should be understood that the embodiments of the present application may be utilized in other aspects and may include structural or logical changes not depicted in the accompanying drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the application is defined by the appended claims. For example, it should be understood that disclosures in connection with a described method may equally apply to a corresponding apparatus or system for performing the described method, and vice versa. For example, if one or more specific method steps are described, the corresponding apparatus may include one or more units, such as functional units, to perform one or more of the described method steps (eg, one unit performs one or more steps) , or units, each of which performs one or more of the steps), even if such unit or units are not explicitly described or illustrated in the figures. On the other hand, if, for example, a specific apparatus is described based on one or more units, such as functional units, the corresponding method may contain a step to perform the functionality of the one or more units (eg, a step to perform the one or more units) functionality, or steps, each of which performs the functionality of one or more of the plurality of units), even if such one or more steps are not explicitly described or illustrated in the figures. Further, it is to be understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other unless expressly stated otherwise.

The image detection method described in the present application can be applied to the field of computer vision, where the image input into the image detection model needs to be processed to output a scene suitable for the image detection model to perform image detection.

Please refer to FIG. 1 , which shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 1 , the electronic device 100 may be a user equipment (User Equipment, UE), such as various types of devices such as a mobile phone, a tablet computer, a smart screen, or an image capturing device. In addition, the electronic device 100 may also be a vehicle. A camera 101 may be provided in the electronic device 100 for capturing image data. In addition, the electronic device 100 may also include or be integrated into a module, chip, chip set, circuit board or component in the electronic device, and the chip or chip set or the circuit board equipped with the chip or chip set can be driven by necessary software Work. The electronic device 100 includes one or more processors, such as a central processing unit (CPU, Central Processing Unit/Processor), an image signal processor (ISP, Image Signal Processor) 103, an AI processor 104, and the like. Optionally, the one or more processors can be integrated in one or more chips, and the one or more chips can be regarded as a chipset, when one or more processors are integrated in the same chip The chip is also called a system on a chip (SOC). In addition to the one or more processors, the electronic device 100 also includes one or more other necessary components, such as memory and the like.

The camera device 101 shown in FIG. 1 may be a monocular camera. Alternatively, the camera device 101 may further include multi-camera cameras, and these cameras may be physically combined in one camera device, or may be physically separated into multiple camera devices. Multiple image data are collected at the same time by the multi-eye camera, and can be processed according to the image data to obtain an image. In a specific implementation, the camera 101 can collect image data in real time, or collect image data periodically. The period is such as 3s, 5s, 10s and so on. The camera device 101 may also collect images in other ways, which are not specifically limited in this embodiment of the present application. The camera device 101 can preprocess the collected image data to generate images to be detected, and provide them to the parameter prediction device 102 and the ISP 103 respectively. The generated image to be detected may be a color image, such as a red green blue (RGB, Red Green Blue) image.

The parameter prediction device 102 shown in FIG. 1 is configured to receive the image to be detected from the camera device 101 . Then, the parameter prediction device 102 predicts the parameters of the image processing algorithm used for processing the image to be detected in the ISP 103 based on the image features of the image to be detected, and generates a parameter prediction value. The parameter prediction apparatus 102 will be introduced below through two possible implementation manners.

In a first possible implementation manner, the parameter prediction apparatus 102 includes an application-specific integrated circuit 1021 and an AI processor 1022, as shown in FIG. 2a. The application-specific integrated circuit 1021 may be dedicated to extracting image features of the image to be detected. The image features may include, but are not limited to, at least one of the following: color features, brightness features, and edge features (for a specific method of feature extraction, refer to the relevant description of the embodiment shown in FIG. 7 ). The application-specific integrated circuit 1021 can be used alone as a component or integrated in other digital logic devices including, but not limited to, a CPU. The AI processor 1022 may include a neural network processor (NPU, Neural-network Processing Unit), a random forest processor, a support vector machine processor and other special purpose processors, including but not limited to a convolutional neural network processor, a tensor processor or neural processing engine. The AI processor 1022 can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU, GPU (Graphics Processing Unit, Graphics Processing Unit) or DSP (Digital Signal Processor, Digital Signal Processor) ). Illustratively, the CPU, GPU and DSP are all processors within a system-on-chip. A pre-trained parameter prediction model can be run in the AI processor 1022 . In this possible implementation manner, the ASIC 1021 extracts image features of the image to be detected, and based on the feature extraction result, generates a feature tensor and provides it to the AI processor 1022 . The parameter prediction model running in the AI processor 1022 predicts the parameters of the image processing algorithm running in the ISP 103 based on the feature tensor, and generates a parameter prediction value and provides it to the ISP 103 .

In a second possible implementation manner, the parameter prediction apparatus 102 may not be provided with an application-specific integrated circuit 1021, but only be provided with an AI processor 1022, as shown in FIG. 2b. The hardware structure of the AI processor 1022 is the same as the hardware structure of the AI processor 1022 shown in FIG. 2a, and details are not repeated here. In this implementation, the AI processor 1022 can also run a pre-trained parameter prediction model. Different from the implementation shown in FIG. 2 a , the parameter prediction model can directly input the image to be detected provided by the first camera device 101 . The parameter prediction model can complete the image feature extraction process and the parameter prediction process of the image to be detected, and then generate the parameter prediction value and provide it to the ISP103.

As shown in the AI processor 1022 shown in FIG. 2a and FIG. 2b, the parameter prediction model run in the AI processor 1022 may be the detection result of the sample image S1 by comparing the annotation information of the sample image S1 and the image detection model, And based on the comparison results, the initial model is trained. The initial model may include, but is not limited to, a neural network model, a support vector machine model, or a random forest model. It should be noted that the parameter prediction model is deployed in the AI process 1022 shown in FIG. 2a or FIG. 2b after the offline-end training is completed. The offline end here can be regarded as a server device or a device for model training. The above-mentioned image detection model is an image detection model running in the AI processor 104 . The method for generating the parameter prediction model may specifically refer to the embodiment shown in FIG. 8 .

The ISP 103 shown in FIG. 1 can set up multiple hardware modules or run necessary software programs to process the image data. The ISP103 can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU, GPU or DSP. The ISP 103 may perform a plurality of image processing processes, which may include, but are not limited to, tone mapping, contrast enhancement, edge enhancement, noise reduction, color correction, and the like.

It should be noted that the ISP 103 executes the above-mentioned multiple image processing processes by running the image processing algorithm. The image processing algorithms may include, but are not limited to, tone mapping algorithms, contrast enhancement algorithms, edge enhancement algorithms, noise reduction algorithms, color correction algorithms, and the like. Each image processing process in the above-mentioned multiple image processing processes can be regarded as an independent image processing process, and thus, the image processing algorithm for executing each image processing process can be regarded as independent. Based on this, the ISP 103 may include multiple logic modules. For example, it includes but is not limited to: tone mapping module, contrast enhancement module, edge enhancement module, color correction module, etc. Each logic module is used to perform an image processing procedure. Each logic module may use its own specific hardware structure, and multiple logic modules may also share a set of hardware structures, which is not limited in this embodiment of the present application. The one or more image processing procedures are typically performed sequentially. For example, after the image acquired by the camera 101 is provided to the ISP 103, processing processes such as tone mapping, contrast enhancement, edge enhancement, color correction, etc. may be sequentially performed. It should be noted that this embodiment of the present application does not limit the sequence of the image processing processes performed by the ISP 103 , for example, the contrast enhancement may be performed first, and then the tone mapping may be performed.

In the embodiment of the present application, in the image processing algorithm run by the ISP 103, the values of some parameters are adjustable. For example, the Gamma function parameter γ in the tone mapping algorithm; the contrast threshold parameter in the contrast enhancement algorithm; the edge enhancement factor α in the edge enhancement algorithm; the spatial domain Gaussian kernel parameter σs and the pixel value domain Gaussian kernel parameter in the noise reduction algorithm σr. The detailed description of each adjustable parameter in the image processing algorithm refers to the embodiment shown in FIG. 9 . The parameter prediction values generated by the parameter prediction model are the values of these adjustable parameters in the image processing algorithm. The parameter prediction value output by the parameter prediction model may include multiple values, such as target brightness parameter value, target saturation parameter value, filter kernel parameter value, edge enhancement factor α value, spatial domain Gaussian kernel parameter value and pixel value domain Gaussian kernel parameter value. That is to say, the ISP 103 can obtain the value of the adjustable parameter of the image processing algorithm executing each image processing flow from the parameter prediction model, and update the value of the corresponding adjustable parameter to the parameter prediction value generated by the parameter prediction model. Then, the ISP 103 performs image processing on the image input by the first camera 101 by using the image processing algorithm after parameter adjustment.

The AI processor 104 shown in FIG. 1 may include a special-purpose neural processor such as a Neural-network Processing Unit (NPU), including but not limited to a convolutional neural network processor, a tensor processor, or a neural processing unit. engine. The AI processor can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU, GPU or DSP. The AI processor 104 may run an image detection model, and the image detection model is obtained by training a deep neural network based on the sample image S2. This image detection model can perform specific detection tasks. The specific detection task may include, but is not limited to, labeling of detection frames, recognition of target objects, prediction of confidence levels, prediction of motion trajectories of target objects, or image segmentation, and the like. It should be noted that the image detection model is deployed in the AI processor 104 shown in FIG. 1 after the offline-end training is completed. The offline end here can be regarded as a server device or a device for model training.

It should be noted that the AI processor 1022 and the AI processor 104 in the parameter prediction apparatus 102 may be co-located in the same AI processor, and the AI processor may run both the parameter prediction model and the image detection model. In addition, the AI processor 1022 and the AI processor 104 may also be independent processors.

Generally, the image detection model executed in the AI processor 104 usually performs image detection based on features such as brightness, color, and texture of the image. Among them, features such as brightness, color, and texture are usually reflected by indicators such as the dynamic range, noise level, and contrast of the image. For multiple images representing the same object, when the dynamic range, noise, contrast and other indicators of the multiple images are different, the characteristics such as brightness, color and texture of the multiple images are also different. There will also be differences in the detection results of the multiple images by the image detection model. Usually, the image detection model is deployed in the terminal device after the offline training is completed, and the experts who train the image detection model are usually different from the users who use the image detection model. That is to say, when users use the image detection model, they usually cannot obtain the indicators (such as dynamic range, noise, contrast) of the sample images used to train the image detection model, which leads to practical use, such as shown in Figure 3. In the autonomous driving scene of 2000, when the external environment changes (such as night, fog, glare, dust, etc.), the images collected by the camera device in terms of dynamic range, contrast, noise and other indicators are different from those used to train the image detection model. Compared with the sample image, it will change. This leads to a large deviation between the detection results output by the image detection model and the real results, which reduces the detection accuracy of the image detection model. In the embodiment of the present application, a parameter prediction model is set, and the parameter prediction model is used to predict the dynamic range, noise level, local contrast and other indicators of the image based on the image characteristics of the image to be detected, and the parameter prediction value is provided to the ISP103, so that the ISP103 runs The image processing algorithm processes the image to be detected based on the predicted value of the parameter, so that the dynamic range, noise level, local contrast and other indicators of the image input to the image detection model can be compared with the dynamic range of the sample image used for training the image detection model. The noise level, local contrast and other indicators are the same or similar, thereby improving the accuracy of the detection results of the image detection model.

In traditional computer vision technology, in actual use, in order to improve the generalization ability of the image detection model, that is, it can detect images collected in special environments (such as night, fog, glare, dust, etc.), usually in training image detection models. Artificially change the image shift, viewing angle, size, brightness, color and other features (or a combination of the above changes) in the samples of the sample to generate a training data set with a larger number and a more discrete sample distribution. Re-training or adjustment, which increases the cost of training the image detection model; in addition, when the capacity of the image detection model is limited and fixed, the generalization ability and detection accuracy of the image detection model cannot be achieved at the same time. In order to improve the image detection model The generalization ability of the image detection model will lead to the degradation of the detection performance of the image detection model, so the accuracy of the detection results output by the image detection model is still not ideal. By setting the parameter prediction model in the embodiment of the present application, it is not necessary to make any changes to the parameters of the image detection model. Compared with the traditional technology, the time and computing power overhead required for training the image detection model can be saved, and the The image detection model can maintain a high detection accuracy; in addition, since the parameter prediction model predicts the parameters of the image processing algorithm, it does not need to perform the image detection process, so the image detection model does not need to prepare high-quality reference images during the training process. , the training can be completed with fewer training samples, which simplifies the training process of the parameter prediction model.

Taking an automatic driving scenario as an example, the following describes the application scenario of the embodiment of the present application in more detail in conjunction with the schematic structural diagram of the electronic device 100 shown in FIG. 1 . Please refer to FIG. 3 , which shows a schematic structural diagram of a vehicle 300 provided by an embodiment of the present application.

Components coupled to or included in vehicle 300 may include control system 10 , propulsion system 20 , and sensor system 30 . It should be understood that the vehicle 300 may also include more systems, which will not be repeated here. The control system 10 may be configured to control the operation of the vehicle 300 and its components. The parameter prediction device 102, ISP 103 and AI processor 104 shown in FIG. 1 may be set in the control system 10. In addition, the control system 10 may also include devices such as a central processing unit, a memory, etc., and the memory is used to store the operating conditions of each processor. required commands and data. Propulsion system 20 may be used for vehicle 300 to provide powered motion, which may include, but is not limited to, an engine/motor, energy source, transmission, and wheels. The sensor system 30 may include, but is not limited to, a global positioning system, an inertial measurement unit, a lidar sensor, or a millimeter-wave radar sensor. The camera 101 shown in FIG. 1 may be disposed in the sensor system 30 . The components and systems of vehicle 300 may be coupled together through a system bus, network, and/or other connection mechanism to operate in interconnection with other components within and/or outside of their respective systems. In specific work, various components in the vehicle 300 cooperate with each other to realize various automatic driving functions. The automatic driving function may include, but is not limited to, blind spot detection, parking assist or lane change assist, and the like.

In the process of implementing the above automatic driving function, the camera device 101 may periodically collect and generate images to be detected, and provide the images to be detected to the parameter prediction device 102 . The parameter prediction device 102 predicts the parameters of the image processing algorithm for processing the image to be detected based on the image features of the image to be detected, and generates a parameter prediction value and provides it to the ISP 103. The ISP103 adjusts the size of the adjustable parameters in the image processing algorithm run by the ISP103 based on the parameter prediction value, and then uses the image processing algorithm after parameter adjustment to perform multiple image processing processes to process the image to be detected, and converts it into the image to be detected by the AI processor 104. The images recognized or calculated by the running image detection model can be provided to the AI processor 104, so that the AI processor 104 can perform reasoning or detection of a specific task and generate detection results. Other components in the control system 10 (eg, a CPU that executes decisions) control other devices or components to perform autonomous driving functions based on the detection results of the AI processor 104 .

Taking the structure of the parameter prediction apparatus 102 shown in FIG. 2 a as an example, the interaction flow between the components in the electronic device 100 shown in FIG. 1 will be described below. Referring to FIG. 4 , FIG. 4 schematically shows an interaction process 400 between components in each electronic device 100 as described in FIG. 1 provided by an embodiment of the present application. The process 400 includes the following steps:

In step 401 , the camera 101 collects image data, processes the collected image data, and generates an image A.

In step 402, the camera 101 provides the image A to the ASIC 1021 and the ISP 103, respectively.

Step 403, the ASIC 1021 performs feature extraction on the image A, and generates a feature tensor based on the feature extraction result. Step 404 , the ASIC 1021 provides the feature tensor to the AI processor 1022 .

Step 405, the parameter prediction model running in the AI processor 1022 predicts the parameters of the image processing algorithm running in the ISP 103 based on the feature tensor, and generates a parameter prediction value. In step 406, the AI processor 1022 provides the parameter prediction value to the ISP 103.

Step 407 , the ISP 103 adjusts the parameters of the image processing algorithm based on the parameter prediction value, and uses the image processing algorithm after parameter adjustment to process the image A to generate the image B. In step 408 , the ISP 103 provides the image B to the AI processor 104 .

Step 409, the image detection model running in the AI processor 104 detects the image B, and generates a detection result.

It should be understood that the steps or operations of the interaction process 400 shown in FIG. 4 are only examples, and other operations or variations of the respective operations in FIG. 4 may also be performed in this embodiment of the present application. In addition, the embodiments of the present application may further include more or less steps than those shown in FIG. 4 . For example, when the parameter prediction device 102 has the structure shown in FIG. 2 b , steps 403 and 404 may be omitted, and the camera device 101 directly provides the image A to the AI processor 1022 .

In the electronic device 100 shown in FIG. 1 , the image output by the imaging device 101 is a color image. In other possible implementation manners, the image output by the camera device 101 may be an image in a raw image format (RAW Image Format). Since the parameter prediction device 102 cannot perform feature extraction and prediction based on the RAW image, when the image captured by the camera device 101 is a RAW image, the ISP 103 can also perform the process of converting the RAW image into a color image. At this time, the camera device 101 can provide the RAW image to the ISP 103 , and the ISP 103 processes the RAW image to generate a color image and then provide it to the parameter prediction device 102 , as shown in FIG. 5 . Image processing algorithms for converting RAW images into color images in ISP 103 may include, but are not limited to, dark current correction algorithms, lens shading correction algorithms, and demosaicing algorithms. The structures of the parameter prediction apparatus 102 and the AI processor 104 shown in FIG. 5 are the same as the structures of the parameter prediction apparatus 102 and the AI processor 104 shown in FIG. It is not repeated here.

Based on the schematic structural diagram of the electronic device 100 shown in FIG. 5 , please refer to FIG. 6 . FIG. 6 shows an interaction process 600 between components in each electronic device 100 as shown in FIG. 5 provided by an embodiment of the present application. In the interaction process 600 shown in FIG. 6 , steps 403 to 409 are the same as steps 403 to 409 shown in FIG. 4 , and details are not repeated here. Wherein, steps 401-402 in the interaction process 400 shown in FIG. 4 are replaced by steps 4011-4013 shown in FIG. 6:

In step 4011, the camera 101 collects image data to obtain a RAW image. In step 4012, the camera 101 provides the RAW image to the ISP 103. Step 4013 , the image processing algorithm run by the ISP 103 processes the RAW image to generate an image A. In step 4014 , the ISP 103 provides the image A to the ASIC 1021 .

Based on the embodiments shown in FIG. 1 to FIG. 6 , taking the extracted image features as color features, brightness features, and edge features as an example, and in conjunction with FIG. 7 , the feature extraction of the images to be detected described in the above embodiments The method for generating feature tensors is described in detail.

In this embodiment of the present application, the color feature may be represented by a red-blue (RB) chromaticity histogram, the luminance feature may be represented by a luminance-differential luminance histogram, and the edge feature may be represented by a two-dimensional edge feature map. The ASIC 1021 described in the above embodiments can sequentially construct the RB chromaticity histogram _{H C} , the luminance-differential luminance histogram HL and the edge feature map _IE of the image to be detected. The resolutions of the RB chrominance histogram _{H C} , the luminance-difference luminance histogram HL and the edge feature map _IE are equal. Then, the ASIC 1021 can combine the RB chrominance histogram _{H C} , the luminance-difference luminance histogram HL and the edge feature map _IE to generate a K×K×3 feature tensor _TF , where K×K are the resolutions of the RB chrominance histogram _{H C} , the luminance-difference luminance histogram HL and the edge feature map _IE respectively, namely

T _F ＝concatenate _{axis＝channel} (H _C ,H _L ,I _E ) Formula (1)

The construction method of the RB chrominance histogram _{H C} , the luminance-difference luminance histogram HL and the edge feature map _IE will be described in detail below.

Construction of RB chromaticity histogram H _C :

For each pixel in the image to be detected, calculate the ratio of the pixel value of the green channel to the pixel value of the red channel (G/R), denoted as r, and in the same way, calculate the ratio of the pixel value of the green channel to the pixel value of the blue channel ( G/B), denoted as b. The column height H _C (r ₀ , b ₀ ) at the position (r ₀ , b ₀ ) in the chromaticity histogram indicates that the chromaticity coordinates (r, b) in the input image correspond to (r ₀ , b _{0 )} The number of pixels in the chromaticity range of

where Δr and Δb represent the widths of a single histogram in the r and b directions in the chromaticity histogram, respectively, and ‖·‖ represents the number of elements in the set. By traversing each pixel in the input image and calculating its coordinates in the rb plane, the construction of the RB chromaticity histogram H _C can be completed. The resolution of the RB chromaticity histogram H _C depends on the size of the statistical interval of the rb plane and the width of the histogram, and is usually set to 128×128.

Construction of luminance-differential luminance histogram _HL :

The brightness value Y of each pixel in the image to be detected is defined as the weighted summation result of the three-channel pixel values of the red channel pixel value R, the green channel pixel value G, and the blue channel pixel value B at the pixel, that is

Y=0.299R+0.587G+0.114B Formula (3)

The differential luminance value D of each pixel is defined as the average absolute difference between the luminance value at this pixel and the luminance values of 8 pixels in its area, namely

Among them, (x, y) represents the pixel at a certain coordinate point in the image, Y(x, y) represents the brightness of the pixel at the coordinate point, (x+i, y+j) represents the difference between (x, y) in the image ) The pixel distance at this coordinate point is the pixel of (i, j) coordinate, and Y(x+i, y+j) represents the pixel brightness of (x+i, y+j) this coordinate point. The differential luminance value at a pixel usually reflects how different the pixel is from surrounding pixels. The column height H _L (Y _{0 ,} D ₀ ) at the position (Y ₀ , D ₀ ) in the luminance histogram indicates that the luminance of the input image and the differential luminance coordinates (Y, D) fall within (Y ₀ , D ₀ ) The number of pixels in the corresponding range, that is

where ΔY and ΔD represent the widths of a single histogram in the luminance histogram in the direction of luminance value Y and differential luminance value D, respectively. By traversing each pixel in the input image and calculating its coordinates in the YD plane, the _HL construction of the luminance-difference luminance histogram can be completed. The resolution of the luminance-difference luminance histogram _HL depends on the size of the statistical interval of the YD plane and the width of the histogram. The resolution of the luminance-differential luminance histogram HL needs to be set to be equal to the resolution of the RB chrominance histogram _{H C.}

Construction of edge feature map _IE :

The brightness channel of the image to be detected is generated by the Canny edge detection operator, morphological closure operation and spatial downsampling, namely

表示形态学中的膨胀操作，

表示腐蚀操作，B表示用于闭合操作的结构元素，通常选用圆盘结构(disk structure)即可。在构建图像边缘特征的过程中，Canny边缘检测算子用于检测出图像中的物体边缘区域；形态学闭合操作用于消除边缘图像中狭窄的间断和长细的鸿沟，消除小的空洞，并填补轮廓线中的断裂；空间下采样用于将边缘图像的分辨率处理至与色度直方图H _C的分辨率及亮度直方图H _L的分辨率相等。 where Resize( ) represents the spatial downsampling operation, Canny( ) represents the Canny edge detection operator,

represents the dilation operation in morphology,

Indicates the corrosion operation, and B represents the structural element used for the closing operation, usually a disk structure can be selected. In the process of constructing image edge features, the Canny edge detection operator is used to detect the edge regions of objects in the image; the morphological closure operation is used to eliminate narrow discontinuities and long thin gaps in the edge image, eliminate small holes, and Breaks in contour lines are filled; spatial downsampling is used to process the edge image resolution to be equal to the resolution of the chrominance histogram _{H C} and the luminance histogram HL.

Based on the embodiment shown in Fig. 1-Fig. 5, the following takes the image processing algorithm including the tone mapping algorithm, the contrast enhancement algorithm, the edge enhancement algorithm and the noise reduction algorithm as an example, combined with Fig. 8, the image processing algorithm running in the ISP103 parameters are described in detail.

Tone mapping is used to receive a linear image with high bit depth, convert the linear image into a nonlinear image, and complete the compression of the image bit depth, and output an 8-bit image. When the global gamma function is used as the tone mapping function, the adjustable parameter in the tone mapping algorithm is the γ parameter; when the logarithmic transformation algorithm is used to compress the dynamic range of the linear image, the adjustable parameter in the tone mapping algorithm is The base of the logarithmic transformation; when using a more complex tone mapping model, such as the retinex model based on the dynamic range response of the human eye, the adjustable parameters in the tone mapping algorithm are the target brightness parameter (key), target saturation parameters (saturation) and filter kernel parameters used to generate the low pass filtered image.

Contrast Enhancement is used to enhance the contrast of an image. Specifically, the CLAHE (contrast limited adaptive histogram equalization, contrast limited adaptive histogram equalization) algorithm can be used to locally adjust the contrast of the image. Adjustable parameters in the CLAHE algorithm include: contrast threshold parameter and sub-image block size for histogram statistics. In a possible implementation manner of the embodiment of the present application, the size of the sub-image block may be fixed, and only the contrast threshold parameter may be adjusted. Further, the size of the sub-image block can be fixed to the size of the input image.

In the edge enhancement of the image, the Y-channel image in the received image is first subjected to Gaussian filtering to obtain a low-pass Y-channel image Y _L ; the difference image between the original Y-channel image and the low-pass Y-channel image Y _L As the high-frequency signal in the image, that is, Y _HF =YY _L , the high-frequency signal usually corresponds to the edge area in the image; by amplifying the intensity of the high-frequency signal and superimposing it into the low-pass Y-channel image Y _L , it can be The image Y _E after edge enhancement is obtained, that is, Y _E =Y _L +α·(YY _L ), where α is an edge enhancement factor, and the degree of image edge enhancement increases with the increase of α. The adjustable parameter in the edge enhancement algorithm is the edge enhancement factor α.

In image noise reduction, the bilateral filter noise reduction algorithm is usually used. In the bilateral filtering noise reduction algorithm, the adjustable parameters may include: a spatial Gaussian kernel parameter σs used to control the relationship between the noise reduction intensity and the spatial distance, and a pixel used to control the relationship between the noise reduction intensity and the response value difference Value domain Gaussian kernel parameter σr.

Please continue to refer to FIG. 9 , which shows a schematic diagram 900 of a system architecture including an electronic device for generating a parameter prediction model provided by an embodiment of the present application.

In FIG. 9 , the system architecture 900 includes a model training device 901 , a storage device 902 and a display device 903 . The storage device 902 may include, but is not limited to, read-only memory or random access memory, and the like. The storage device 902 is used to store the sample image S1. In addition, the storage device 902 may also store executable programs and data of image processing algorithms for performing image processing, executable programs and data for performing image feature extraction, executable programs and data of the parameter prediction model to be trained, and data, and executable programs and data for an image detection model used to perform image detection. The model training device 901 can run the feature extraction algorithm, the executable program and data of the parameter prediction model to be trained, the image processing algorithm and the image detection model, and the model training device 901 can also call the sample image S1, the image processing algorithm from the storage device 902 The executable program and data, the executable program and data for performing image feature extraction, the executable program and data for the parameter prediction model to be trained, and the executable program and data for performing the image detection model for image detection, To debug the parameters of the parameter prediction model. In addition, the model training device 901 may also store the data generated by the operation and the debugging results after each parameter debugging of the parameter prediction model to the storage device 902 . In addition, the model training device 901 and the storage device 902 may also be provided with I/O ports for data interaction with the display device 903 . The user equipment 903 may include a display device such as a screen to mark the sample image S1. Specifically, the model training device 901 may acquire the sample image S1 from the storage device 902 , perform image processing on the sample image S1 and provide the sample image S1 to the display device 903 for presentation in the display device 903 . The user annotates the sample image S1 through the display device 903 , and stores the annotation information of the sample image S1 in the storage device 902 .

Based on the schematic structural diagrams of the electronic device 100 shown in FIGS. 1 and 4 , and the system architecture 900 shown in FIG. 9 , a method for generating a parameter prediction model will be described in detail below with reference to FIG. 10 . Please refer to FIG. 10 , which shows a process 1000 of a method for generating a parameter prediction model provided by an embodiment of the present application. It should be noted that, the execution body of the method for generating a parameter prediction model described in the embodiments of the present application may be the model training device 901 shown in FIG. 9 . As shown in Figure 10, the method for generating a parameter prediction model includes the following steps:

Step 1001 , based on the sample image set, perform feature extraction on the sample image S1 in the sample image set to generate a feature tensor.

The sample image set includes multiple sample images S1 and label information of each sample image S1. The annotation information of the sample image S1 is annotated based on the detection content performed by the image detection model. For example, when the image detection model is used to perform target detection, the annotation information of the sample image S1 may include the target object and the position of the target object in the sample image S1; when the image detection model is used to perform pedestrian intent detection, the sample image S1 The annotation information may include the target object and the action information of the target object.

In this embodiment of the present application, the process of performing feature extraction on the sample image S1 to generate a feature tensor is the same as the process of generating a feature tensor described in the embodiment shown in FIG. description, which will not be repeated here.

Step 1002 , input the feature tensor into the parameter prediction model, and generate the parameters of the image processing algorithm for processing the sample image S1.

In the embodiment of the present application, the image processing algorithm includes, but is not limited to, a tone mapping algorithm, a noise reduction algorithm, a contrast enhancement algorithm, or an edge enhancement algorithm, and the like. Wherein, the parameters of the image processing algorithm specifically include but are not limited to: the target brightness parameter (key), the target saturation parameter (saturation) in the tone mapping algorithm, and the filter kernel parameter used to generate the low-pass filtered image; in the contrast enhancement algorithm The contrast threshold parameter of ; the edge enhancement factor α in the edge enhancement algorithm; the spatial domain Gaussian kernel parameter σs and the pixel value domain Gaussian kernel parameter σr in the noise reduction algorithm.

The parameter prediction model described in the embodiments of the present application may include one of the following: a random forest model, a support vector machine model, or a neural network model. The structure of the parameter prediction model is described in detail below by taking the parameter prediction model as the neural network model as an example. The parameter prediction model can include multi-layer convolutional layers, multi-layer pooling layers and fully connected layers. Among them, the convolution layer is used for feature extraction and convolution operations on the image to generate feature maps; the pooling layer is used to downsample and reduce the dimension of the feature maps; the fully connected layer uses several levels of fully connected neural networks to The feature vector output by the convolutional layer is calculated, and finally the predicted value of the parameter is output. In a possible implementation, each convolutional layer can use a 3×3 convolution kernel, and set the number of channels of the output feature map to twice the input feature map, without changing the spatial resolution of the feature map; Each pooling layer uses a max pooling operator to reduce the spatial resolution of the input feature map to 1/2 the original resolution. Assuming that the feature tensor input to the parameter prediction network is K×K×3, the feature vector of 1×1×3K is generated after the feature tensor is processed by multiple convolutional layers and pooling layers. The fully connected layer calculates the 1×1×3K feature vector, and generates a 1×1×n parameter prediction vector. Each element in the parameter prediction vector corresponds to a configurable parameter in the image processing algorithm.

The work of each convolutional layer in the parameter prediction model can be described by the mathematical expression y=a(W*x+b). Among them, W is the weight, x is the input vector (ie the input neuron), b is the bias data, y is the output vector (ie the output neuron), and a is a constant. From the physical level, the work of each layer in the deep neural network can be understood as completing the transformation from the input space to the output space (that is, the row space of the matrix to the column space) through five operations on the input space (set of input vectors). These five operations include: 1. Dimension up/down; 2. Zoom in/out; 3. Rotate; 4. Translation; 5. "Bend". Among them, the operations of 1, 2, and 3 are completed by W*x, the operation of 4 is completed by +b, and the operation of 5 is implemented by a(). The reason why the word "space" is used here is because the image to be processed is not a single thing, but a type of thing, and space refers to the collection of all individuals of this type of thing. where W is the weight vector, and each value in the vector represents the weight value of a neuron in the convolutional layer of this layer. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how the space is transformed.

Step 1003: Based on the generated parameters of the image processing algorithm for processing the sample image S1, adjust the image processing algorithm, and use the image processing algorithm after parameter adjustment to process the sample image S1 to generate a processed image B.

In step 1004, the image B is detected by using the image detection model, and a detection result is generated.

Here, the image detection model may perform at least one of the following detections: object detection, lane line detection, or pedestrian intent detection, and the like. The image detection model is obtained by training a deep neural network based on the image dataset S1. Among them, the image detection model can be obtained by training using the traditional model training method, which will not be repeated here.

Step 1005, construct a loss function based on the detection result and the labeling information of the sample image S1.

A loss function is constructed based on the error between the detection result of each sample image S1 in the sample image dataset and the annotation information of the sample image S1. The loss function may include, but is not limited to, a mean absolute error (MAE) loss function, a mean square error (MSE) loss function, or a cross entropy function, and the like. The constructed loss function includes parameters to be adjusted in the parameter prediction model. The parameters of the parameter prediction model are the weight matrices of all convolutional layers forming the parameter prediction model (the weight matrix formed by the vectors W of many convolutional layers).

Step 1006, determine whether a preset condition is reached.

The preset conditions here include at least one of the following: the loss value of the preset loss function is less than or equal to the preset threshold; the number of times of iteratively adjusting the parameters of the parameter prediction model is greater than or equal to the preset threshold. When the preset condition is reached, the parameters of the parameter prediction model are saved; when the preset condition is not reached, step 1007 is executed.

Step 1007: Adjust the parameters of the parameter prediction model by using the back-propagation algorithm and the gradient descent algorithm.

The gradient descent algorithm may specifically include, but is not limited to, optimization algorithms such as SGD and Adam. When performing backpropagation based on the preset loss function, the chain rule can be used to calculate the gradient of the preset loss function with respect to each weight matrix in the parameter prediction model.

It should be noted that when the parameters of the image processing algorithm are adjusted by the back-propagation algorithm based on the preset loss function, the parameters in the image detection model are kept unchanged. In addition, when using the machine learning method to adjust the parameters of the parameter prediction model, the image detection algorithms described in the embodiments of the present application all have differentiability, so as to facilitate backpropagation based on the chain rule.

For the training method of the parameter prediction model shown in FIG. 10 , by repeatedly performing steps 1002 to 1007, that is, performing multiple iterative adjustments to the parameters of the parameter prediction model, the parameter prediction model that makes the loss function reach a minimum value can be obtained. optimal parameter value.

With reference to the above embodiments and the related drawings, the embodiments of the present application provide an image detection method, which can be implemented in the electronic device shown in FIG. 1 and FIG. 5 . Figure 11 is a flow chart 1100 of an image detection method provided by an embodiment of the present application, and as shown in Figure 11, the method may include the following steps:

Step 1101, acquiring an image to be detected.

Step 1102 , based on the image features of the image to be detected, use a pre-trained parameter prediction model to predict the parameters of the image processing algorithm used to process the image to be detected, and generate a parameter prediction value.

Step 1103: Perform image processing on the image to be detected by using the image processing algorithm after parameter adjustment to generate a processed image.

Step 1104: Input the processed image into a pre-trained image detection model to generate a detection result.

It can be understood that, in order to realize the above-mentioned functions, the electronic device includes corresponding hardware and/or software modules for executing each function. The present application can be implemented in hardware or in the form of a combination of hardware and computer software in conjunction with the algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functionality for each particular application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

In this embodiment, the above one or more processors may be divided into functional modules according to the foregoing method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. . The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that, the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division manners in actual implementation.

In the case where each functional module is divided according to each function, FIG. 12 shows a possible schematic diagram of the composition of the image detection apparatus 1200 involved in the above embodiment. As shown in FIG. 12 , the image detection apparatus 1200 may include: An acquisition module 1201 , a prediction module 1202 , an adjustment module 1203 , a processing module 1204 and a detection module 1205 . The acquisition module 1201 is configured to acquire the image to be detected; the prediction module 1202 is configured to use a pre-trained parameter prediction model based on the image features of the image to be detected, to process the image for processing the image to be detected. The parameters of the processing algorithm are predicted to generate parameter predicted values; the adjustment module 1203 is configured to adjust the parameters of the image processing algorithm based on the parameter predicted values; the processing module 1204 is configured to use the image processing algorithm adjusted by the parameters Image processing is performed on the to-be-detected image to generate a processed image; the detection module 1205 is configured to input the processed image into a pre-trained image detection model to generate a detection result.

In a possible implementation manner, the parameter prediction model is obtained by comparing the annotation information of the sample image with the detection result of the sample image by the image detection model, and training based on the comparison result.

In a possible implementation manner, the image detection apparatus further includes a parameter prediction model training module (not shown in the figure), and the parameter prediction model training module includes: a comparison sub-module configured to compare the sample image The detection result of the sample image is compared with the annotation information of the sample image to obtain the comparison result; the adjustment sub-module is configured to iteratively adjust the parameters of the parameter prediction model based on the comparison result; the saving sub-module is configured to When the preset condition is satisfied, the parameters of the parameter prediction model are saved.

In a possible implementation manner, the comparison result is an error, and the adjustment sub-module is further configured to: based on the error between the detection result of the sample image and the annotation information of the sample image, construct A target loss function, wherein the target loss function includes parameters to be adjusted in the parameter prediction model; based on the target loss function, the parameters of the image processing algorithm are iteratively adjusted by using a back-propagation algorithm and a gradient descent algorithm.

In a possible implementation manner, the prediction module 1202 is configured to: perform feature extraction on the to-be-detected image to generate a feature tensor; input the feature tensor into the parameter prediction model to generate the feature tensor Parameter predictions.

In a possible implementation manner, the prediction module 1202 is configured to: input the image to be detected into the parameter prediction model to generate the parameter prediction value.

In a possible implementation manner, the image processing algorithm includes at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm, and an image noise reduction algorithm.

In a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence levels, and prediction of motion trajectories of target objects.

The image detection apparatus 1200 provided in this embodiment is configured to execute the image detection method executed by the electronic device 100, and can achieve the same effect as the above-mentioned implementation method. Each module corresponding to FIG. 12 can be implemented by software, hardware or a combination of the two. For example, each module can be implemented in software to drive the parameter prediction device 102, ISP 103 and AI processing in the electronic device 100 shown in FIG. 1 . device 104. Alternatively, each module may include a corresponding processor and a corresponding driver software.

Where an integrated unit is employed, the image detection apparatus 1200 may include at least one processor and memory. Wherein, at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the electronic device 100, for example, it can be used to support the electronic device 100 to perform the steps performed by the above-mentioned modules. The memory may be used to support the execution of the electronic device 100 by storing program codes and data, and the like. The processor may implement or execute various exemplary logic modules described in conjunction with the present disclosure, which may be a combination of one or more microprocessors that implement computing functions, such as, but not limited to, the image signal shown in FIG. 1 . Processor 103 and AI Processor 104. In addition, the microprocessor combination may also include a central processing unit, a controller, and the like. In addition, the processor may include other programmable logic devices, transistor logic devices, or discrete hardware components in addition to the processors shown in FIG. 1 . The memory may include random access memory (RAM) and read only memory (ROM), among others. The random access memory can include volatile memory (such as SRAM, DRAM, DDR (Double Data Rate SDRAM, Double Data Rate SDRAM) or SDRAM, etc.) and non-volatile memory. The RAM can store data (such as image processing algorithms, etc.) and parameters required for the operation of the parameter prediction device 102, ISP 103 and the AI processor 104, intermediate data generated by the operation of the parameter prediction device 102, ISP 103 and the AI processor 104, and the ISP 103 The processed image data, the output result after the AI processor 104 runs, and the like. Executable programs of the parameter prediction device 102 , the ISP 103 and the AI processor 104 may be stored in the read-only memory ROM. Each of the above components can perform their own work by loading an executable program. The executable program stored in the memory can execute the image detection method as described in FIG. 11 .

In the case where each functional module is divided according to each function, FIG. 13 shows a possible schematic diagram of the composition of the apparatus 1300 for training a parameter prediction model involved in the above embodiment. As shown in FIG. The apparatus 1300 for training a parameter prediction model may include: a prediction module 1301 , a first adjustment module 1302 , a processing module 1303 , a detection module 1304 , a comparison module 1305 and a second adjustment module 1306 . The prediction module 1301 is configured to use a parameter prediction model to predict the parameters of the image processing algorithm used to process the sample image based on the image features of the sample image, and generate a parameter prediction value; the first adjustment module 1302 is configured by is configured to adjust the parameters of the image processing algorithm based on the parameter prediction value; the processing module 1303 is configured to perform image processing on the sample image by using the image processing algorithm after parameter adjustment, and generate a processed image; the detection module 1304, input the processed image into a pre-trained image detection model, and generate a detection result; a comparison module 1305, is configured to compare the error between the detection result and the labeling information of the sample image, and obtain a comparison result ; The second adjustment module 1306 is configured to adjust the parameters of the parameter prediction model based on the comparison result; the saving module is configured to save the parameters of the parameter prediction model when a preset condition is satisfied.

In a possible implementation manner, the comparison result is an error, and the second adjustment module 1306 is configured to: construct an objective loss function based on the error between the detection result and the annotation information of the sample image, The target loss function includes parameters to be adjusted in the parameter prediction model; based on the target loss function, the parameters of the parameter prediction model are iteratively adjusted by using a back-propagation algorithm and a gradient descent algorithm.

In a possible implementation manner, the prediction module 1301 is configured to: perform feature extraction on the sample image to generate a feature tensor; input the feature tensor into the parameter prediction model to generate the parameter prediction value .

In a possible implementation, the prediction module 1301 is configured to: input the sample image into the parameter prediction model to generate the parameter prediction value.

Where an integrated unit is employed, the apparatus 1300 for training a parameter prediction model may include at least one processor and storage device. Wherein, at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the model training device 901 as shown in FIG. step. The memory may be used to support the execution of the model training device 901 by storing program codes and data, and the like. The processor can implement or execute various exemplary logic modules described in conjunction with the disclosure of the present application, which can be one or more microprocessor combinations that implement computing functions, including but not limited to a central processing unit and a controller, etc. . In addition, the processor may also include other programmable logic devices, transistor logic devices, or discrete hardware components, or the like. The memory may include random access memory (RAM), read only memory ROM, and the like. The random access memory can include volatile memory (such as SRAM, DRAM, DDR (Double Data Rate SDRAM, Double Data Rate SDRAM) or SDRAM, etc.) and non-volatile memory. The RAM may store data (such as image processing algorithms, etc.) and parameters required by the model training device 901 to run, intermediate data generated by the model training device 901 running, and output results after the model training device 901 runs. An executable program of the model training apparatus 901 may be stored in the read-only memory ROM. Each of the above components can perform their own work by loading an executable program. The executable program stored in the memory may perform the method for training a parameter prediction model as described in FIG. 10 .

This embodiment further provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on the computer, the computer executes the above-mentioned related method steps to realize the image detection in the above-mentioned embodiment. The image detection method of the apparatus 1100, or the training method of the parameter prediction model of the apparatus 1300 for training the parameter prediction model in the above-mentioned embodiment.

This embodiment also provides a computer program product, which, when the computer program product runs on a computer, causes the computer to execute the above-mentioned relevant steps, so as to realize the image detection method of the image detection apparatus 1100 in the above-mentioned embodiment, or to realize the above-mentioned embodiment. A training method of a parameter prediction model of the apparatus 1300 for training a parameter prediction model in .

Wherein, the computer-readable storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, for the beneficial effect that can be achieved, reference may be made to the corresponding method provided above. The beneficial effects will not be repeated here.

From the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated by different The function module is completed, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (which may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. that can store program codes. medium.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application. scope.

Claims (20)

An image detection method, comprising: Obtain the image to be detected; Based on the image features of the to-be-detected image, using a pre-trained parameter prediction model, predict the parameters of the image processing algorithm used to process the to-be-detected image, and generate a parameter prediction value; based on the parameter prediction value, adjusting the parameters of the image processing algorithm; Image processing is performed on the to-be-detected image by using the image processing algorithm after parameter adjustment to generate a processed image; The processed images are input into a pre-trained image detection model to generate detection results. The image detection method according to claim 1, wherein the parameter prediction model is obtained by comparing the annotation information of the sample image with the detection result of the sample image by the image detection model, and is trained based on the comparison result. The image detection method according to claim 2, wherein the parameter prediction model is obtained by training the following steps: comparing the detection result of the sample image with the annotation information of the sample image to obtain the comparison result; Iteratively adjust the parameters of the parameter prediction model based on the comparison results; When the preset condition is satisfied, the parameters of the parameter prediction model are saved. The image detection method according to claim 3, wherein the comparison result is an error, and the iteratively adjusting the parameters of the parameter prediction model based on the comparison result comprises: constructing a target loss function based on the error between the detection result of the sample image and the annotation information of the sample image, wherein the target loss function includes parameters to be adjusted in the parameter prediction model; Based on the objective loss function, the parameters of the image processing algorithm are iteratively adjusted using a back-propagation algorithm and a gradient descent algorithm. The image detection method according to any one of claims 1-4, characterized in that, based on the image features of the to-be-detected image, a pre-trained parameter prediction model is used to process the image-to-be-detected image. The parameters of the image processing algorithm are predicted, and the predicted values of the parameters are generated, including: performing feature extraction on the to-be-detected image to generate a feature tensor; The feature tensor is input to the parameter prediction model, and the parameter prediction value is generated. The image detection method according to any one of claims 1-4, characterized in that, based on the image features of the to-be-detected image, a pre-trained parameter prediction model is used to process the image-to-be-detected image. The parameters of the image processing algorithm are predicted, and the predicted values of the parameters are generated, including: The to-be-detected image is input into the parameter prediction model, and the parameter prediction value is generated. The image detection method according to any one of claims 1-6, wherein the image processing algorithm comprises at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm and an image noise reduction algorithm. The image detection method according to any one of claims 1-7, wherein the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence, Prediction of the motion trajectory of the target object. An image detection device, characterized in that it includes: an acquisition module, configured to acquire the image to be detected; A generation module, configured to predict the parameters of the image processing algorithm for processing the image to be detected by using a pre-trained parameter prediction model based on the image features of the image to be detected, and generate a parameter prediction value; an adjustment module configured to adjust a parameter of the image processing algorithm based on the parameter predicted value; a processing module, configured to perform image processing on the to-be-detected image by using an image processing algorithm after parameter adjustment, to generate a processed image; A detection module configured to input the processed image to a pre-trained image detection model to generate a detection result. The image detection device according to claim 9, wherein the parameter prediction model is obtained by comparing the annotation information of the sample image with the detection result of the sample image by the image detection model, and is trained based on the comparison result. The image detection device according to claim 10, wherein the image detection device further comprises a parameter prediction model training module, the parameter prediction model training module comprising: a comparison submodule, configured to compare the detection result of the sample image with the annotation information of the sample image to obtain the comparison result; an adjustment submodule configured to iteratively adjust parameters of the parameter prediction model based on the comparison results; The saving sub-module is configured to save the parameters of the parameter prediction model when the preset condition is satisfied. The image detection device according to claim 11, wherein the comparison result is an error, and the adjustment sub-module is further configured to: constructing a target loss function based on the error between the detection result of the sample image and the annotation information of the sample image, wherein the target loss function includes parameters to be adjusted in the parameter prediction model; Based on the objective loss function, the parameters of the image processing algorithm are iteratively adjusted using a back-propagation algorithm and a gradient descent algorithm. The image detection device according to any one of claims 9-12, wherein the generation module is configured to: performing feature extraction on the to-be-detected image to generate a feature tensor; The feature tensor is input to the parameter prediction model, and the parameter prediction value is generated. The image detection device according to any one of claims 9-12, wherein the generation module is configured to: The to-be-detected image is input into the parameter prediction model, and the parameter prediction value is generated. The image detection device according to any one of claims 9-14, wherein the image processing algorithm comprises at least one of the following: a tone mapping algorithm, a contrast enhancement algorithm, an image edge enhancement algorithm and an image noise reduction algorithm. The image detection device according to any one of claims 9-14, wherein the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence, Prediction of the motion trajectory of the target object. An electronic device, comprising: a camera device for acquiring an image to be detected; A parameter prediction device, configured to predict parameters of an image processing algorithm used to process the image to be detected by using a pre-trained parameter prediction model based on the image features of the image to be detected, and generate a parameter prediction value; an image signal processor, configured to adjust the parameters of the image processing algorithm based on the parameter prediction value, and perform image processing on the to-be-detected image by using the image processing algorithm after parameter adjustment to generate a processed image; The artificial intelligence processor is used for inputting the processed image to a pre-trained image detection model to generate a detection result. An image detection device, characterized in that it includes: one or more processors and memories; the memory is coupled to the processor, the memory for storing one or more programs; The one or more processors are configured to run the one or more programs to implement the method of any one of claims 1-8. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by at least one processor, is used to implement the method according to any one of claims 1-8. method. A computer program product, characterized in that, when the computer program product is executed by at least one processor, it is used to implement the method according to any one of claims 1-8.

Images