CN117014701A

CN117014701A - Terminal equipment and camera method

Info

Publication number: CN117014701A
Application number: CN202210453822.XA
Authority: CN
Inventors: 钟炯生; 胡杨
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-04-27
Filing date: 2022-04-27
Publication date: 2023-11-07

Abstract

The application discloses a terminal device and an image pickup method, which can pick up images of objects with good quality for objects with various colors on the surfaces, and is beneficial to extracting the surface information of the objects with enough accuracy. The terminal device of the present application comprises: the camera comprises a plurality of light sources positioned at different positions, and each light source in the plurality of light sources can emit light rays with a plurality of wavelengths, namely, each light source in the plurality of light sources is a multi-wavelength light source. When a user needs to shoot a target object, a request can be issued to a processor of the terminal device, so the processor can control the camera to start a plurality of light sources in the camera according to a preset sequence based on the request from the user, so that the light sources emit light with a certain wavelength to the target object in turn, and the camera is enabled to shoot the target object, and the image of the target object is successfully acquired.

Description

Terminal equipment and shooting method

Technical Field

The embodiment of the application relates to the technical field of artificial intelligence (artificial intelligence, AI), in particular to terminal equipment and a shooting method.

Background

With the continuous development of manufacturing industry, users and enterprises have higher requirements on product quality, and the product has good appearance, namely good surface besides the service performance meeting the requirements. Therefore, after the product is manufactured, the surface of the product is often required to be evaluated to determine whether the surface of the product is qualified, that is, whether the surface of the product has defects is detected, so that the quality of the product is ensured.

At present, when it is required to determine whether the surface of a certain product is acceptable, surface information of the product (which may also be referred to as three-dimensional information of the product, including information such as reflectivity and gaussian curvature of the product) may be obtained first, then the surface information of the product is analyzed by using a certain mathematical method, and based on the analysis result, it may be determined that the product is an acceptable product or an unacceptable product.

However, in the above-described evaluation mode, the surface information of the product may be acquired by some algorithm. In the algorithm, the image of the product can be acquired through a camera according to a certain shooting mode, so that the surface information of the product can be acquired based on the image of the product. However, cameras referred to herein are typically deployed with an LED white light source provided with a bead that emits blue light, and the bead surface is coated with a yellow phosphor. Then, when the light source is started, the lamp bead can emit blue light, and the blue light can emit yellow light after exciting the fluorescent powder, and white light is formed after mixing. Therefore, the imaging effect of the light source on the products with blue and yellow surfaces is better than that of the products with other colors on the surfaces, so that the surface information of the products with other colors on the surfaces cannot be accurately acquired.

Disclosure of Invention

The embodiment of the application provides a terminal device and an imaging method, which can shoot images of objects with good quality for objects with various colors on the surface, and is beneficial to extracting the surface information of the objects with enough accuracy.

A first aspect of the embodiments of the present application provides a terminal device, which is typically an execution device used locally by a user, so that the terminal device may be referred to as a local execution device. The local execution device comprises: the camera comprises a plurality of light sources positioned at different positions, wherein each light source in the plurality of light sources can emit light rays with a plurality of wavelengths, namely, each light source in the plurality of light sources is a multi-wavelength light source.

When a user needs to shoot a target object, a request can be issued to a processor of a local execution device, so the processor can control the camera to start a plurality of light sources in the camera according to a preset sequence based on the request from the user (for example, 5 light sources are arranged in the camera, the processor can control the camera to sequentially start the 5 light sources according to the sequence of the 5 light sources), so that the plurality of light sources alternately emit light rays with a certain wavelength to the target object, and the camera can complete shooting of the target object, so that images of the target object are successfully acquired.

In this way, after the processor of the local execution device obtains the image of the target object, the surface evaluation of the target object can be completed based on the image of the target object, that is, whether the surface of the target object has a defect or not is detected.

The camera of the local execution device comprises a plurality of light sources, each light source in the plurality of light sources is a multi-wavelength light source, so before the processor of the local execution device controls the camera to shoot objects with various colors on the surface, the processor can adjust the wavelength of light rays emitted by the plurality of light sources in the camera to adapt to the objects with various colors on the surface, and therefore the imaging effect of the camera on the objects with various colors on the surface is high enough, namely, the image of the object obtained by shooting by the camera has good quality, and the camera is favorable for extracting the surface information of the objects with accurate enough.

In one possible implementation, the plurality of light sources are arranged in an array of light sources, and an included angle between two adjacent light sources in the plurality of light sources is a preset included angle. In the foregoing implementation manner, the plurality of light sources in the camera may form a light source array, so the camera may sequentially activate different light sources to shoot the object at different positions (azimuth angles) to obtain a plurality of images of the object.

In one possible implementation, the camera satisfies at least one of the following: the exposure time of the camera is smaller than or equal to a preset first threshold value, the brightness of the plurality of light sources is larger than or equal to a preset second threshold value, the signal to noise ratio of the image shot by the camera is larger than or equal to a preset third threshold value, and the definition of the image shot by the camera of the camera is larger than or equal to a preset fourth threshold value, wherein the signal to noise ratio is determined based on the wavelength of the light rays emitted by the plurality of light sources. In the foregoing implementation manner, before controlling the camera of the local execution device to capture the object (including the target object and the reference object), the processor of the local execution device may perform parameter adjustment on the camera of the local execution device, where the parameter adjustment operation includes at least one of: (1) The exposure time of the camera is adjusted so that the exposure time of the camera is smaller than or equal to a preset first threshold, wherein the preset first threshold can be understood as an exposure time threshold required by a service environment in which the object is located. (2) The brightness of the plurality of light sources in the camera is adjusted such that the brightness of the plurality of light sources is greater than or equal to a preset second threshold, wherein the preset second threshold may be understood as a light source brightness threshold required for an imaging effect of an object of sufficient quality. (3) The wavelengths of the light rays emitted by the plurality of light sources in the camera are adjusted so that the signal to noise ratio of the image of the reference object shot by the camera is greater than or equal to a preset third threshold, wherein the preset third threshold can be understood as an image signal to noise ratio threshold required for the imaging effect of the object with sufficient quality. (4) The sharpness of the image of the reference object captured by the camera is adjusted such that the sharpness of the image of the reference object captured by the camera is greater than or equal to a preset fourth threshold, wherein the preset fourth threshold may be understood as a sharpness threshold required for an imaging effect of an object of sufficient quality. After the parameters of the camera are adjusted, the local execution equipment can shoot the object through the adjusted camera, and the obtained image of the object can be shot, so that the camera has excellent image quality.

In one possible implementation, the processor is further configured to: acquiring surface information of a target object based on an image of the target object, wherein the surface information of the target object comprises reflectivity of the target object and/or Gaussian curvature of the target object; processing the surface information of the target object through the feature extraction model to obtain the features of the target object; and determining that the target object is a surface-qualified object or a surface-unqualified object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-qualified object. In the foregoing implementation manner, after obtaining the image of the target object, the processor of the local execution device may calculate the image of the target object, so as to obtain surface information (stereo information) of the target object, where the content included in the surface information of the target object is one of the following three cases: the surface information of the target object includes only the reflectivity of the target object, or the surface information of the target object includes only the gaussian curvature of the target object, or the surface information of the target object includes both the reflectivity and the gaussian curvature of the target object. After the surface information of the target object is obtained, the processor of the local execution device can input the surface information of the target object into a feature extraction model (trained neural network model) so as to process the surface information of the target object through the feature extraction model, thereby obtaining the features of the target object. After determining the characteristics of the target object, the processor of the local execution device may compare the characteristics of the target object with the characteristics of the reference object, thereby determining whether the surface of the target object is acceptable, i.e. whether the surface of the target object has defects. In the foregoing implementation manner, no matter what type of product the target object is, the processor of the local execution device may utilize the feature extraction model to process the surface information of the target object, so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surface, so as to determine whether the surface of the target object is qualified, so that the new evaluation manner performs feature extraction on various objects through the neural network model, and can avoid the involvement of artificial factors.

In one possible implementation, the processor is configured to calculate an image of the target object based on a photometric stereo algorithm, so as to obtain surface information of the target object. In the foregoing implementation, the processor of the local execution device may calculate the image of the target object using the mathematical model provided by the photometric stereo algorithm, so as to accurately obtain the surface information of the target object.

In one possible implementation, the processor is configured to: obtaining statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object; calculating the characteristics of the target object to obtain statistics of the characteristics of the target object, wherein the statistics of the characteristics of the target object comprise mean vectors of the characteristics of the target object and covariance matrixes of the characteristics of the target object; calculating statistics of the characteristics of the target object and statistics of the characteristics of the reference object to obtain scores of the target object; if the score of the target object is larger than a preset fifth threshold value, determining that the target object is an object with unqualified surface; and if the score of the target object is smaller than or equal to a preset fifth threshold value, determining that the target object is the object with qualified surface. In the foregoing implementation, the processor of the local execution device may receive statistics of features of the reference object from the remote execution device and use the statistics as a unified reference for object surface assessment. Then, after obtaining the characteristics of the target object, the processor of the local execution device may calculate the characteristics of the target object, thereby obtaining statistics of the characteristics of the target object. After obtaining statistics of the features of the target object, the processor of the local execution device may calculate the statistics of the features of the target object and the statistics of the features of the reference object, thereby obtaining a score of the target object. After the score of the target object is obtained, the processor of the local execution device can detect whether the score of the target object is larger than a preset fifth threshold value, if the score of the target object is larger than the preset fifth threshold value, the processor of the local execution device can determine that the target object is an object with a unqualified surface, and if the score of the target object is smaller than or equal to the preset fifth threshold value, the processor of the local execution device can determine that the target object is an object with a qualified surface.

In one possible implementation, the processor is configured to: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; and calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object. In the foregoing implementation manner, the processor of the local execution device may control the camera to capture the reference object, thereby acquiring an image of the reference object, extract surface information of the reference object based on the image of the reference object, process the surface information of the reference object through the feature extraction model, thereby acquiring features of the reference object, and finally calculate statistics of the features of the reference object, and use the statistics of the features of the reference object as the same reference standard for performing surface recognition on various objects.

In one possible implementation, the processor is configured to: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; an image of the reference object is transmitted. In the foregoing implementation manner, the local execution device may be communicatively connected to the remote execution device, and after the processor of the local execution device acquires the image of the reference object, the processor of the local execution device may send the image of the reference object to the remote execution device, so that the remote execution device processes the image of the reference object, thereby obtaining statistics of features of the reference object, and then sends the statistics to the processor of the local execution device. Therefore, the remote execution device can provide a unified reference standard (namely, statistics of characteristics of reference objects) for carrying out surface recognition on various objects for the local execution device so as to jointly realize surface recognition on the objects by combining the local execution device, the number of hardware devices required by a system formed by the remote execution device and the local execution device is often small, the hardware cost and the customization difficulty required by the system deployment are reduced, the remote execution device can be opened for various users, and the remote execution device does not need to bring out excessively high professional requirements for the users, so that the remote execution device can be widely applied to various business scenes.

In one possible implementation, the score of the target object is determined based on a difference between the mean vector of the features of the target object and the mean vector of the features of the reference object, and a matrix obtained by fusing the covariance matrix of the features of the target object with the covariance matrix of the features of the reference object. In the foregoing implementation manner, the processor of the local execution device may calculate a difference between the mean vector of the feature of the target object and the mean vector of the feature of the reference object, then fuse the covariance matrix of the feature of the target object with the covariance matrix of the feature of the reference object, and then calculate a distance between the difference and the fused matrix, where the distance may be used as a score of the target object. In this way, the processor of the local execution device can accurately obtain the score of the target object.

In one possible implementation, the processor is further configured to: among the plurality of regions of the target object, a region having a score greater than a preset fifth threshold value is determined as a region of the target object having a surface failure. In the foregoing implementation manner, after determining that the target object is an object with a surface that is not acceptable, the processor of the local execution device may further consider a plurality of points in the surface of the target object as a plurality of areas of the target object, and since the score of the target object includes the scores of the plurality of areas of the target object, the processor of the local execution device may determine, among the plurality of areas of the target object, an area with a score greater than a preset fifth threshold as an area with a surface that is not acceptable in the target object, thereby accurately determining which part of the area of the surface of the target object has a defect.

In one possible implementation, the processor is further configured to: the target object is determined to be a new reference object. In the foregoing implementation, after determining that the target object is a surface-qualified object, the processor of the local execution device may further determine the target object as a new reference object, so the processor of the local execution device may update statistics of features of the new reference object to statistics of features of the original reference object, to obtain updated statistics. When a user needs to make a surface assessment of a new target object, the processor of the local execution device may determine whether the surface of the new target object is acceptable based on the updated statistics and statistics of the characteristics of the new target object. It can be seen that the embodiment of the present application further provides a feedback mechanism, which enables the local execution device to update the unified reference standard for implementing the object surface assessment.

In one possible implementation, the processor is further configured to: a reference update request is sent, the reference update request being used to indicate the target object. In the foregoing implementation manner, after determining that the target object is a surface-qualified object, the processor of the local execution device may further send a reference object update request to the remote execution device, so that the remote execution device determines the target object indicated by the reference object update request as a new reference object. Since the reference update request may contain statistics of the characteristics of the target object, the remote execution device may update the statistics of the characteristics of the new reference object to the statistics of the characteristics of the original reference object, resulting in updated statistics. Finally, the remote execution device may send the updated statistics to the local execution device, such that the processor of the local execution device replaces the statistics of the features of the original reference object with the updated statistics. When a user needs to make a surface assessment of a new target object, the processor of the local execution device may determine whether the surface of the new target object is acceptable based on the updated statistics and statistics of the characteristics of the new target object. It can be seen that the embodiment of the present application further provides a feedback mechanism, which enables the remote execution device to continuously update the unified reference standard for implementing the object surface assessment in the local execution device side.

A second aspect of the embodiment of the present application provides an image capturing method, where the method is implemented by a terminal device, i.e., a local execution device, where the local execution device includes: the device comprises a processor and a camera, wherein the processor is connected with the camera, the camera comprises a plurality of light sources positioned at different positions, and each light source in the plurality of light sources can emit light rays with a plurality of wavelengths, and the method comprises the following steps: and controlling the camera to start a plurality of light sources according to a preset sequence so as to shoot the target object and obtain an image of the target object.

The camera of the local execution device applied by the method comprises a plurality of light sources, each light source in the plurality of light sources is a multi-wavelength light source, so before the processor of the local execution device controls the camera to shoot objects with various colors on the surface, the processor can adjust the wavelength of light rays emitted by the plurality of light sources in the camera to adapt to the objects with various colors on the surface, thereby ensuring that the imaging effect of the camera on the objects with various colors on the surface is high enough, namely, the image of the object shot by the camera has good quality, and the method is favorable for extracting the surface information of the object with enough accuracy.

In one possible implementation manner, the method further includes, after controlling the camera to activate the plurality of light sources in a preset order to capture the target object and obtain an image of the target object: acquiring surface information of a target object based on an image of the target object, wherein the surface information of the target object comprises reflectivity of the target object and/or Gaussian curvature of the target object; processing the surface information of the target object through the feature extraction model to obtain the features of the target object; and determining that the target object is a surface-qualified object or a surface-unqualified object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-qualified object.

In one possible implementation, acquiring surface information of the target object based on the image of the target object includes: and calculating the image of the target object based on a photometric stereo algorithm to obtain the surface information of the target object.

In one possible implementation, determining that the target object is a surface-eligible or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object includes: obtaining statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object; calculating the characteristics of the target object to obtain statistics of the characteristics of the target object, wherein the statistics of the characteristics of the target object comprise mean vectors of the characteristics of the target object and covariance matrixes of the characteristics of the target object; calculating statistics of the characteristics of the target object and statistics of the characteristics of the reference object to obtain scores of the target object; if the score of the target object is larger than a preset fifth threshold value, determining that the target object is an object with unqualified surface; and if the score of the target object is smaller than or equal to a preset fifth threshold value, determining that the target object is the object with qualified surface.

In one possible implementation, obtaining statistics of features of the reference includes: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; and calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object.

In one possible implementation, the method further includes: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; an image of the reference object is transmitted.

In one possible implementation, the score of the target object is determined based on a difference between the mean vector of the features of the target object and the mean vector of the features of the reference object, and a matrix obtained by fusing the covariance matrix of the features of the target object with the covariance matrix of the features of the reference object.

In one possible implementation, after determining that the target object is a surface-failed object, the method further includes: among the plurality of regions of the target object, a region having a score greater than a preset fifth threshold value is determined as a region of the target object having a surface failure.

In one possible implementation, after determining that the target object is a surface-qualified object, the method further includes: the target object is determined to be a new reference object.

In one possible implementation, after determining that the target object is a surface-qualified object, the method further includes: a reference update request is sent, the reference update request being used to indicate the target object.

A third aspect of an embodiment of the present application provides an object surface evaluation apparatus, the apparatus being deployable in a processor of a local execution device, the processor being connected to a camera of the local execution device, the camera comprising a plurality of light sources located at different positions, each of the plurality of light sources being capable of emitting light of a plurality of wavelengths, the apparatus comprising:

the first acquisition module is used for controlling the camera to start a plurality of light sources according to a preset sequence so as to shoot a target object and obtain an image of the target object; a second acquisition module for acquiring surface information of a target object based on an image of the target object, the surface information of the target object including a reflectivity of the target object and/or a gaussian curvature of the target object; the processing module is used for processing the surface information of the target object through the feature extraction model to obtain the features of the target object; the first determining module is used for determining that the target object is a surface-qualified object or a surface-unqualified object based on the characteristics of the target object and the characteristics of the reference object, and the reference object is a surface-qualified object.

From the above device, it can be seen that: after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object comprises a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

In one possible implementation manner, the second acquisition module is configured to calculate an image of the target object based on a photometric stereo algorithm, so as to obtain surface information of the target object.

In one possible implementation, the camera satisfies at least one of the following: the exposure time of the camera is smaller than or equal to a preset first threshold value, the brightness of the plurality of light sources is larger than or equal to a preset second threshold value, the signal to noise ratio of the image shot by the camera is larger than or equal to a preset third threshold value, and the definition of the image shot by the camera of the camera is larger than or equal to a preset fourth threshold value, wherein the signal to noise ratio is determined based on the wavelength of the light rays emitted by the plurality of light sources.

In one possible implementation, the plurality of light sources are arranged in an array of light sources, and an included angle between two adjacent light sources in the plurality of light sources is a preset included angle.

In one possible implementation manner, the first determining module is configured to: obtaining statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object; calculating the characteristics of the target object to obtain statistics of the characteristics of the target object, wherein the statistics of the characteristics of the target object comprise mean vectors of the characteristics of the target object and covariance matrixes of the characteristics of the target object; calculating statistics of the characteristics of the target object and statistics of the characteristics of the reference object to obtain scores of the target object; if the score of the target object is larger than a preset fifth threshold value, determining that the target object is an object with unqualified surface; and if the score of the target object is smaller than or equal to a preset fifth threshold value, determining that the target object is the object with qualified surface.

In one possible implementation manner, the first determining module is configured to: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; and calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object.

In one possible implementation, the apparatus further includes: the third acquisition module is used for controlling the camera to sequentially start a plurality of light sources so as to shoot the reference object and obtain an image of the reference object; and the second transmitting module is used for transmitting the image of the reference object.

In one possible implementation, the apparatus further includes: and the second determining module is used for determining the area with the score larger than a preset fifth threshold value as the area with the unqualified surface in the target object in a plurality of areas of the target object.

In one possible implementation, the apparatus further includes: and a third determining module for determining the target object as a new reference object.

In one possible implementation, the apparatus further includes: the first sending module is used for sending a reference object updating request which is used for indicating the target object.

A fourth aspect of the embodiments of the present application provides a server deployed on a side remote from a user, and thus may be referred to as a remote execution device, the remote execution device including a memory and a processor; the memory stores code and the processor is configured to execute the code, and when the code is executed, the remote execution device may perform the steps of: acquiring an image of a reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object; statistics of features of the reference object are transmitted.

From the above described device it can be seen that: the remote execution device can provide a unified reference standard (namely statistics of characteristics of reference objects) for carrying out surface recognition on various objects for the local execution device so as to jointly realize surface recognition on the objects by combining the local execution device, the number of hardware devices required by a system formed by the local execution device and the local execution device is often less, the hardware cost and the customization difficulty required by the system deployment are reduced, the system can be opened for various users, and the system does not need to put forward excessively high professional requirements on the users, so that the system can be widely applied to various business scenes.

In one possible implementation, after sending the statistic of the feature of the reference object, the method further includes: acquiring a reference object updating request; the target object indicated by the reference update request is determined to be a new reference object.

A fifth aspect of an embodiment of the present application provides a method for obtaining statistics, where the method is implemented by a remote execution device, and the method includes: acquiring an image of a reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object; statistics of features of the reference object are transmitted.

From the above method, it can be seen that: the remote execution device can provide a unified reference standard (namely statistics of characteristics of reference objects) for carrying out surface recognition on various objects for the local execution device so as to jointly realize surface recognition on the objects by combining the local execution device, the number of hardware devices required by a system formed by the local execution device and the local execution device is often less, the hardware cost and the customization difficulty required by the system deployment are reduced, the system can be opened for various users, and the system does not need to put forward excessively high professional requirements on the users, so that the system can be widely applied to various business scenes.

A sixth aspect of an embodiment of the present application provides a statistic acquisition apparatus deployed in a processor of a remote execution device, the apparatus comprising: the first acquisition module is used for acquiring an image of a reference object; a second acquisition module for acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including a reflectivity of the reference object and/or a gaussian curvature of the reference object; the processing module is used for processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; the computing module is used for computing the characteristics of the reference object to obtain statistics of the characteristics of the reference object, wherein the statistics of the characteristics of the reference object comprise mean vectors of the characteristics of the reference object and covariance matrixes of the characteristics of the reference object; and the sending module is used for sending the statistic of the characteristics of the reference object.

From the above device, it can be seen that: the remote execution device can provide a unified reference standard (namely statistics of characteristics of reference objects) for carrying out surface recognition on various objects for the local execution device so as to jointly realize surface recognition on the objects by combining the local execution device, the number of hardware devices required by a system formed by the local execution device and the local execution device is often less, the hardware cost and the customization difficulty required by the system deployment are reduced, the system can be opened for various users, and the system does not need to put forward excessively high professional requirements on the users, so that the system can be widely applied to various business scenes.

In one possible implementation, the apparatus further includes: the third acquisition module is used for acquiring a reference object update request; and the determining module is used for determining the target object indicated by the reference object updating request as a new reference object.

A seventh aspect of the embodiments of the present application provides a model training apparatus, the apparatus comprising a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the model training apparatus performs the steps of: acquiring an image of an object to be processed; acquiring surface information of an object to be processed based on an image of the object to be processed, wherein the surface information of the object to be processed comprises reflectivity of the object to be processed and/or Gaussian curvature of the object to be processed; processing the surface information of the object to be processed through the model to be trained to obtain the characteristics of the object to be processed; acquiring target loss based on the characteristics of the object to be processed and the real characteristics of the object to be processed, wherein the target loss is used for indicating the difference between the characteristics of the object to be processed and the real characteristics of the object to be processed; and updating parameters of the model to be trained based on the target loss until the model training condition is met, and obtaining the feature extraction model.

The feature extraction model obtained by training the device can be deployed in local execution equipment and remote execution equipment, so that the local execution equipment realizes the function of extracting the features of the target object in the process of identifying the surface of the target object. Specifically, after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

In one possible implementation, acquiring surface information of the object to be processed based on the image of the object to be processed includes: and calculating an image of the object to be processed based on a photometric stereo algorithm to obtain the surface information of the object to be processed.

An eighth aspect of the embodiment of the present application provides a model training method, which further includes: acquiring an image of an object to be processed; acquiring surface information of an object to be processed based on an image of the object to be processed, wherein the surface information of the object to be processed comprises reflectivity of the object to be processed and/or Gaussian curvature of the object to be processed; processing the surface information of the object to be processed through the model to be trained to obtain the characteristics of the object to be processed; acquiring target loss based on the characteristics of the object to be processed and the real characteristics of the object to be processed, wherein the target loss is used for indicating the difference between the characteristics of the object to be processed and the real characteristics of the object to be processed; and updating parameters of the model to be trained based on the target loss until the model training condition is met, and obtaining the feature extraction model.

The feature extraction model obtained by training the method can be deployed in local execution equipment and remote execution equipment, so that the local execution equipment realizes the function of extracting the features of the target object in the process of identifying the surface of the target object. Specifically, after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

A ninth aspect of the embodiment of the present application provides a model training apparatus, which further includes: the first acquisition module is used for acquiring an image of an object to be processed; the second acquisition module is used for acquiring surface information of the object to be processed based on the image of the object to be processed, wherein the surface information of the object to be processed comprises the reflectivity of the object to be processed and/or the Gaussian curvature of the object to be processed; the processing module is used for processing the surface information of the object to be processed through the model to be trained to obtain the characteristics of the object to be processed; the third acquisition module is used for acquiring target loss based on the characteristics of the object to be processed and the real characteristics of the object to be processed, wherein the target loss is used for indicating the difference between the characteristics of the object to be processed and the real characteristics of the object to be processed; and the updating module is used for updating parameters of the model to be trained based on the target loss until the model training condition is met, so as to obtain the feature extraction model.

The feature extraction model obtained by training the device can realize the function of extracting the features of the target object in the process of carrying out surface recognition on the target object. Specifically, after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

In one possible implementation manner, the second acquisition module is configured to calculate an image of the object to be processed based on a photometric stereo algorithm, so as to obtain surface information of the object to be processed.

A tenth aspect of the embodiments of the present application provides a circuit system comprising a processing circuit configured to perform the method according to the second aspect, any one of the possible implementations of the second aspect, the fifth aspect, any one of the possible implementations of the fifth aspect, the eighth aspect or any one of the possible implementations of the eighth aspect.

An eleventh aspect of the embodiments of the present application provides a chip system, the chip system comprising a processor for invoking a computer program or computer instructions stored in a memory to cause the processor to perform the method according to the second aspect, any one of the possible implementations of the second aspect, the fifth aspect, any one of the possible implementations of the fifth aspect, the eighth aspect or any one of the possible implementations of the eighth aspect.

In one possible implementation, the processor is coupled to the memory through an interface.

In one possible implementation, the system on a chip further includes a memory having a computer program or computer instructions stored therein.

A twelfth aspect of the embodiments of the present application provides a computer storage medium storing a computer program which, when executed by a computer, causes the computer to implement a method as in the second aspect, any one of the possible implementations of the second aspect, the fifth aspect, any one of the possible implementations of the fifth aspect, the eighth aspect or any one of the possible implementations of the eighth aspect.

A thirteenth aspect of the embodiments of the present application provides a computer program product storing instructions that, when executed by a computer, cause the computer to implement a method as described in the second aspect, any one of the possible implementations of the second aspect, the fifth aspect, any one of the possible implementations of the fifth aspect, the eighth aspect or any one of the possible implementations of the eighth aspect.

In the embodiment of the application, the camera of the local execution device comprises a plurality of light sources, and each light source in the plurality of light sources is a multi-wavelength light source, so before the processor of the local execution device controls the camera to shoot objects with various colors on the surface, the processor can adjust the wavelength of light rays emitted by the plurality of light sources in the camera to adapt to the objects with various colors on the surface, thereby ensuring that the imaging effect of the camera on the objects with various colors on the surface is high enough, namely, the image of the object shot by the camera has good quality, and the extraction of the surface information of the object with enough accuracy is facilitated.

Further, after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

Drawings

FIG. 1 is a schematic diagram of a structure of an artificial intelligence main body frame;

FIG. 2a is a schematic diagram of an object surface evaluation system according to an embodiment of the present application;

FIG. 2b is a schematic diagram of another embodiment of an object surface assessment system according to the present application;

FIG. 2c is a schematic diagram of an apparatus for processing an object surface assessment system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a system 100 architecture according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of an object surface evaluation method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a light source array according to an embodiment of the present application;

fig. 6 is a schematic flow chart of adjusting a camera according to an embodiment of the present application;

FIG. 7 is a schematic flow chart of a model training method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a structure of an object surface evaluation apparatus according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a statistic acquiring device according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a model training apparatus according to an embodiment of the present application;

FIG. 11 is a schematic structural diagram of an execution device according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a training apparatus according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a chip according to an embodiment of the present application.

Detailed Description

The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which embodiments of the application have been described in connection with the description of the objects having the same attributes. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

With the continuous development of manufacturing industry, a large number of industrial products are produced every day. The requirements of users and enterprises on the quality of products are higher and higher, and the products have good appearance, namely good surface besides the service performance meeting the requirements. Therefore, after a product is manufactured, it is often required to evaluate the surface of the product to determine whether the surface of the product is acceptable, that is, to detect whether the surface of the product has defects (for example, whether the surface of metal has scratches, spots and holes, whether the surface of paper has indentations, whether the surface of glass is mingled with, broken and stained, etc.), thereby ensuring the quality of the product.

Currently, when it is required to determine whether the surface of a certain product is acceptable, surface information of the product (which may also be referred to as three-dimensional information of the product, including information such as reflectivity and gaussian curvature of the product) may be obtained first, and then the surface information of the product is analyzed by a certain mathematical method (for example, a binarization analysis method, a connected domain analysis method, a linear detection analysis method, etc.), so that the product may be determined to be an acceptable product or an unacceptable product based on the analysis result.

However, the mathematical method used in the above-mentioned evaluation method is usually selected based on expert experience, involves human factors, and is considered on one side, and the method cannot be applied to complex and changeable business scenarios, i.e. facing various types of products, and cannot detect defects of all types of products, and may misjudge some products with unqualified surfaces as products with qualified surfaces. It follows that the surface assessment of the product achieved based on this approach tends to be less accurate.

Further, in the above evaluation mode, the surface information of the product can be obtained by photometric stereo method. Specifically, in the algorithm, an image of the product can be acquired through a camera according to a certain shooting mode, so that surface information of the product can be acquired based on the image of the product. However, cameras referred to herein are typically deployed with an LED white light source provided with a bead that emits blue light, and the bead surface is coated with a yellow phosphor. Then, when the light source is started, the lamp bead can emit blue light, and the blue light can emit yellow light after exciting the fluorescent powder, and white light is formed after mixing. Therefore, the imaging effect of the light source on the products with blue and yellow surfaces is better than that of the products with other colors on the surfaces, so that the surface information of the products with other colors on the surfaces cannot be accurately acquired.

Furthermore, the above-mentioned evaluation method can be applied to a system including an intelligent terminal, a control device and a remote server (for example, a system including an industrial camera, an industrial personal computer and a cloud server), which not only costs high hardware cost and high customization degree, but also has high professional requirements for users, and in actual business scenarios, the system is often difficult to debug and apply.

In order to solve the above-mentioned problems, an embodiment of the present application provides an object surface evaluation method (the method includes the aforementioned image capturing method and the aforementioned statistic acquisition method), which can be implemented in combination with artificial intelligence (artificial intelligence, AI) technology. AI technology is a technical discipline that utilizes digital computers or digital computer controlled machines to simulate, extend and extend human intelligence, and obtains optimal results by sensing environments, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Data processing using artificial intelligence is a common application of artificial intelligence.

First, the overall workflow of the artificial intelligence system will be described, referring to fig. 1, fig. 1 is a schematic structural diagram of an artificial intelligence subject framework, and the artificial intelligence subject framework is described below in terms of two dimensions, namely, an "intelligent information chain" (horizontal axis) and an "IT value chain" (vertical axis). Where the "intelligent information chain" reflects a list of processes from the acquisition of data to the processing. For example, there may be general procedures of intelligent information awareness, intelligent information representation and formation, intelligent reasoning, intelligent decision making, intelligent execution and output. In this process, the data undergoes a "data-information-knowledge-wisdom" gel process. The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of personal intelligence, information (provisioning and processing technology implementation), to the industrial ecological process of the system.

(1) Infrastructure of

The infrastructure provides computing capability support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the base platform. Communicating with the outside through the sensor; the computing power is provided by a smart chip (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform comprises a distributed computing framework, a network and other relevant platform guarantees and supports, and can comprise cloud storage, computing, interconnection and interworking networks and the like. For example, the sensor and external communication obtains data that is provided to a smart chip in a distributed computing system provided by the base platform for computation.

(2) Data

The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data relate to graphics, images, voice and text, and also relate to the internet of things data of the traditional equipment, including service data of the existing system and sensing data such as force, displacement, liquid level, temperature, humidity and the like.

(3) Data processing

Data processing typically includes data training, machine learning, deep learning, searching, reasoning, decision making, and the like.

Wherein machine learning and deep learning can perform symbolized and formalized intelligent information modeling, extraction, preprocessing, training and the like on data.

Reasoning refers to the process of simulating human intelligent reasoning modes in a computer or an intelligent system, and carrying out machine thinking and problem solving by using formal information according to a reasoning control strategy, and typical functions are searching and matching.

Decision making refers to the process of making decisions after intelligent information is inferred, and generally provides functions of classification, sequencing, prediction and the like.

(4) General capability

After the data has been processed, some general-purpose capabilities can be formed based on the result of the data processing, such as algorithms or a general-purpose system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

(5) Intelligent product and industry application

The intelligent product and industry application refers to products and applications of an artificial intelligent system in various fields, is encapsulation of an artificial intelligent overall solution, and realizes land application by making intelligent information decisions, and the application fields mainly comprise: intelligent terminal, intelligent transportation, intelligent medical treatment, autopilot, smart city etc.

Next, several application scenarios of the present application are described.

Fig. 2a is a schematic structural diagram of an object surface assessment system according to an embodiment of the present application, where the object surface assessment system includes a user device and a data processing device. The user equipment comprises terminal equipment such as an industrial camera, a mobile phone, a personal computer or an information processing center. The user device initiates the request by the user, so the user device may jointly implement the object surface assessment in conjunction with the data processing device based on the user's request.

The data processing device may be a device or a server having a data processing function, such as a cloud server, a web server, an application server, and a management server. The data processing device can complete object surface assessment together with the user device at the request of the user device (for example, the data processing device can receive an image of an object from the user device through an interactive interface, and then perform image processing in modes of surface information acquisition, machine learning, deep learning, searching, reasoning, decision making and the like through a memory for storing data and a processor link for data processing to obtain the characteristics of the object). The memory in the data processing device may be a generic term comprising a database storing the history data locally, either on the data processing device or on another network server.

In the object surface assessment system shown in fig. 2a, the user device and the data processing device may together achieve a surface assessment of the target object. Specifically, the user device may acquire the reference object input/selected by the user, and after capturing an image of the reference object, the user device may transmit the image of the reference object to the data processing device, so that the data processing device extracts surface information of the reference object with respect to the image of the reference object, and performs a feature extraction application on the surface information of the reference object, thereby obtaining features of the reference object. Further, the user device may acquire the target object (i.e., the object to be evaluated) input/selected by the user, and after the user device captures an image of the target object, the user device may extract surface information of the target object with respect to the image of the target object, and perform feature extraction application on the surface information of the target object, thereby obtaining features of the target object. The user device may then determine whether the surface of the target object is acceptable based on the characteristics of the target object and the characteristics of the reference object, thereby completing the surface evaluation of the target object.

In fig. 2a, the user device and the data processing device may jointly implement the object surface assessment method according to an embodiment of the present application.

Fig. 2b is a schematic diagram of another structure of an object surface evaluation system according to an embodiment of the present application, in fig. 2b, if a user device has a processor with enough computing power, the user device may implement the functions implemented by the foregoing data processing device, and the user device may directly obtain an input from a user and directly process the input by hardware of the user device, where a specific process is similar to that of fig. 2a, and reference may be made to the above description.

In the object surface assessment system shown in fig. 2b, the user device itself may perform the surface assessment of the target object. Specifically, the user device may acquire a reference object input/selected by the user, and after capturing an image of the reference object, the user device may extract surface information of the reference object with respect to the image of the reference object, and perform a feature extraction application on the surface information of the reference object, thereby obtaining features of the reference object. Further, the user device may acquire the target object (i.e., the object to be evaluated) input/selected by the user, and after the user device captures an image of the target object, the user device may extract surface information of the target object with respect to the image of the target object, and perform feature extraction application on the surface information of the target object, thereby obtaining features of the target object. The user device may then determine whether the surface of the target object is acceptable based on the characteristics of the target object and the characteristics of the reference object, thereby completing the surface evaluation of the target object.

In fig. 2b, the user device itself may perform the object surface assessment method according to the embodiment of the present application.

Fig. 2c is a schematic diagram of an apparatus for evaluating the surface of an object according to an embodiment of the present application.

The user device in fig. 2a and 2b may be specifically the execution device 301 or the execution device 302 deployed locally in fig. 2c, and the data processing device in fig. 2a may be specifically the execution device 210 deployed remotely in fig. 2c, where the data storage system 250 may store data to be processed of the execution device 210, and the data storage system 250 may be integrated on the execution device 210, or may be disposed on a cloud or other network server.

The processor of each execution device may be deployed with a neural network model or other model that has been trained (e.g., a model based on a support vector machine), and perform a special effect extraction application on the surface information of the object using these models, thereby obtaining a corresponding processing result, that is, the feature of the object.

Fig. 3 is a schematic diagram of a system 100 architecture according to an embodiment of the present application, where in fig. 3, the system includes an execution device 110 (such as the aforementioned data processing device) and an execution device 140 (such as the aforementioned user device), a training device 120, and a data acquisition device 160. Wherein the executing device 110 configures an input/output (I/O) interface 112 for data interaction with external devices. For example, the user may issue a first request to the execution device 140 to cause the execution device 140 to collect data (e.g., an image of a reference object) and to input the data to the I/O interface 112.

During preprocessing of the data from the execution device 140 by the execution device 110 (for example, obtaining of surface information of the reference object) or performing relevant processing such as calculation by the calculation module 111 of the execution device 110 (for example, feature extraction of the surface information of the reference object by the feature extraction model), the execution device 110 may call the data, the code, etc. in the data storage system 150 for corresponding processing, or may store the data (for example, the feature of the reference object outputted by the feature extraction model and the information indicating the feature of the reference object), the instruction, etc. obtained by the corresponding processing in the data storage system 150.

The I/O interface 112 then returns the processing results (e.g., information indicating the characteristics of the reference object) to the execution device 140.

After receiving the processing result, the execution device 140 may wait for the user's request again. When the performing device 140 receives the second request from the user, data (e.g., an image of the target object) may be collected again. At this time, the executing device 140 may perform preprocessing on the acquired data (for example, acquiring surface information of the target object), and perform calculation by using its own calculating module (not shown in fig. 3) (for example, performing feature extraction on the surface information of the target object by using the feature extraction model), so as to obtain corresponding processing results (for example, features of the target object output by the feature extraction model and information indicating the features of the target object).

Finally, the execution device 140 may further process and compare the processing results from the execution device 110 and the processing results obtained by itself, thereby obtaining a response (e.g., notifying the user about whether the surface of the target object is acceptable, etc.) that can be fed back to the user.

It should be noted that the training device 120 may generate, based on different training data, a corresponding target model/rule (e.g., a feature extraction model) for different targets or different tasks (e.g., to obtain features of an object), where the corresponding target model/rule may be used to achieve the targets or to perform the tasks, thereby providing the user with a desired result. Training device 120 trains to obtain target models/rules, which may be deployed in execution device 110 and execution device 140. Wherein the training data may be stored in database 130 and derived from training samples collected by data collection device 160.

In the case shown in fig. 3, the execution device 140 may provide a visual interface to the user to receive a request entered manually by the user and activate the own camera upon the request to capture data (e.g., an image of a reference object or an image of a target object, etc.) by the camera. If the executing device 140 is required to automatically transmit data, the user's authorization needs to be obtained, so the user can set the corresponding rights in the executing device 140. The user may view the processing result returned by the execution device 110 at the execution device 140, and the specific presentation form may be a specific manner of display, sound, action, and the like. The execution device 140 may also be used as a data collection end to collect training samples and store the training samples in the database 130, and of course, the training samples may also be collected without going through the execution device 140.

It should be noted that fig. 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among devices, apparatuses, modules, etc. shown in the drawing is not limited in any way, for example, in fig. 3, the data storage system 150 is an external memory with respect to the execution device 110, and in other cases, the data storage system 150 may be disposed in the execution device 110.

The embodiment of the application also provides a chip, which comprises the NPU. The chip may be provided in the execution device 110 and the execution device 140 as shown in fig. 3 to complete the calculation work of the calculation module. The chip may also be provided in the training device 120 as shown in fig. 3 to complete the training work of the training device 120 and output the target model/rule.

The neural network processor NPU is mounted as a coprocessor to a main central processing unit (central processing unit, CPU) (host CPU) which distributes tasks. The core part of the NPU is an operation circuit, and the controller controls the operation circuit to extract data in a memory (a weight memory or an input memory) and perform operation.

In some implementations, the arithmetic circuitry includes a plurality of processing units (PEs) internally. In some implementations, the operational circuit is a two-dimensional systolic array. The arithmetic circuitry may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the operational circuitry is a general-purpose matrix processor.

For example, assume that there is an input matrix a, a weight matrix B, and an output matrix C. The arithmetic circuit takes the data corresponding to the matrix B from the weight memory and caches the data on each PE in the arithmetic circuit. The operation circuit takes the matrix A data and the matrix B from the input memory to perform matrix operation, and the obtained partial result or the final result of the matrix is stored in an accumulator (accumulator).

The vector calculation unit may further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector computation unit may be used for network computation of non-convolutional/non-FC layers in a neural network, such as pooling, batch normalization (batch normalization), local response normalization (local response normalization), and the like.

In some implementations, the vector computation unit can store the vector of processed outputs to a unified buffer. For example, the vector calculation unit may apply a nonlinear function to an output of the arithmetic circuit, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit generates a normalized value, a combined value, or both. In some implementations, the vector of processed outputs can be used as an activation input to an arithmetic circuit, for example for use in subsequent layers in a neural network.

The unified memory is used for storing input data and output data.

The weight data is transferred to the input memory and/or the unified memory directly by the memory cell access controller (direct memory access controller, DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

And a bus interface unit (bus interface unit, BIU) for implementing interaction among the main CPU, the DMAC and the instruction fetch memory through a bus.

The instruction fetching memory (instruction fetch buffer) is connected with the controller and used for storing instructions used by the controller;

and the controller is used for calling the instruction which refers to the cache in the memory and controlling the working process of the operation accelerator.

Typically, the unified memory, input memory, weight memory, and finger memory are On-Chip (On-Chip) memories, and the external memory is a memory external to the NPU, which may be a double data rate synchronous dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM), a high bandwidth memory (high bandwidth memory, HBM), or other readable and writable memory.

Because the embodiments of the present application relate to a large number of applications of neural networks, for convenience of understanding, related terms and related concepts of the neural networks related to the embodiments of the present application will be described below.

(1) Neural network

The neural network may be composed of neural units, which may refer to an arithmetic unit having xs and intercept 1 as inputs, and the output of the arithmetic unit may be:

where s=1, 2, … … n, n is a natural number greater than 1, ws is the weight of xs, and b is the bias of the neural unit. f is an activation function (activation functions) of the neural unit for introducing a nonlinear characteristic into the neural network to convert an input signal in the neural unit to an output signal. The output signal of the activation function may be used as an input to the next convolutional layer. The activation function may be a sigmoid function. A neural network is a network formed by joining together a number of the above-described single neural units, i.e., the output of one neural unit may be the input of another. The input of each neural unit may be connected to a local receptive field of a previous layer to extract features of the local receptive field, which may be an area composed of several neural units.

The operation of each layer in a neural network can be described by the mathematical expression y=a (wx+b): the operation of each layer in a physical layer neural network can be understood as the transformation of input space into output space (i.e., row space to column space of the matrix) is accomplished by five operations on the input space (set of input vectors), including: 1. dimension increasing/decreasing; 2. zoom in/out; 3. rotating; 4. translating; 5. "bending". Wherein operations of 1, 2, 3 are completed by Wx, operation of 4 is completed by +b, and operation of 5 is completed by a (). The term "space" is used herein to describe two words because the object being classified is not a single thing, but rather a class of things, space referring to the collection of all individuals of such things. Where W is a weight vector, each value in the vector representing a weight value of a neuron in the layer neural network. The vector W determines the spatial transformation of the input space into the output space described above, i.e. the weights W of each layer control how the space is transformed. The purpose of training the neural network is to finally obtain a weight matrix (a weight matrix formed by a plurality of layers of vectors W) of all layers of the trained neural network. Thus, the training process of the neural network is essentially a way to learn and control the spatial transformation, and more specifically to learn the weight matrix.

Since it is desirable that the output of the neural network is as close as possible to the value actually desired, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network with the actually desired target value and then according to the difference between the two (of course, there is usually an initialization process before the first update, that is, the pre-configuration parameters of each layer in the neural network), for example, if the predicted value of the network is higher, the weight vector is adjusted to be predicted to be lower, and the adjustment is continued until the neural network can predict the actually desired target value. Thus, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is a loss function (loss function) or an objective function (objective function), which are important equations for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function is, the larger the difference is, and the training of the neural network becomes the process of reducing the loss as much as possible.

(2) Back propagation algorithm

The neural network can adopt a Back Propagation (BP) algorithm to correct the parameter in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model is smaller and smaller. Specifically, the input signal is transmitted forward until the output is generated with error loss, and the parameters in the initial neural network model are updated by back propagation of the error loss information, so that the error loss is converged. The back propagation algorithm is a back propagation motion that dominates the error loss, and aims to obtain parameters of the optimal neural network model, such as a weight matrix.

The method provided by the application is described below from the training side of the neural network and the application side of the neural network.

The model training method provided by the embodiment of the application relates to the processing of a data sequence, and can be particularly applied to methods such as data training, machine learning, deep learning and the like, and intelligent information modeling, extraction, preprocessing, training and the like of symbolizing and formalizing training data (for example, images of objects to be processed in the application) are performed, so that a trained neural network (such as a feature extraction model in the application) is finally obtained; in addition, the object surface evaluation method provided by the embodiment of the present application may use the above trained neural network, and input data (for example, an image of a reference object and an image of a target object in the present application, etc.) into the trained neural network, so as to obtain output data (for example, characteristics of the reference object and characteristics of the target object in the present application, etc.). It should be noted that, the model training method and the object surface evaluation method provided by the embodiments of the present application are applications based on the same concept, and may be understood as two parts in a system or two stages of an overall process: such as a model training phase and a model application phase.

Fig. 4 is a schematic flow chart of an object surface evaluation method according to an embodiment of the present application, and it should be noted that in the embodiment shown in fig. 4, the local execution device and the remote execution device are used to jointly implement the object surface estimation for schematic description. As shown in fig. 4, the method includes:

401. the local execution equipment acquires an image of a reference object, wherein the reference object is an object with qualified surface.

In this embodiment, in order to enable the local execution device to have the function of evaluating the object surface, the user may first acquire an image of a reference object through the local execution device, and it should be noted that the reference object is usually an object with a qualified surface selected in advance by the user, and may be used as a reference standard for performing surface evaluation on the target object (to-be-evaluated object) later.

Specifically, the local execution device is usually deployed with a camera, so the local execution device can shoot the reference object through the camera, thereby acquiring an image of the reference object. Wherein, local execution device's camera can set up a plurality of light sources, and these a plurality of light sources possess two kinds of characteristics below:

(1) The multiple light sources are disposed at different positions, so that when different light sources in the camera are lighted in turn, the camera can irradiate the object through the light sources at different positions (azimuth angles), thereby acquiring an image of the object, and the image of the object obtained based on the mode can be used for acquiring surface information (three-dimensional information) of the object. Further, the plurality of light sources may be arranged in a form that the plurality of light sources may be arranged in a light source array in which an included angle between two adjacent light sources is a preset included angle. For example, as shown in fig. 5 (fig. 5 is a schematic diagram of a light source array provided in an embodiment of the present application), a camera provided with a local execution device includes 4 light sources, where the 4 light sources are uniformly arranged on an annular base and are arranged in a circumferential array, and an included angle between two adjacent 2 light sources in the 4 light sources is 90 °.

(2) Each light source in the plurality of light sources can emit light rays (including visible light and invisible light) with a plurality of wavelengths, namely, each light source is a multi-wavelength light source, so that for objects with various colors on the surfaces, the camera can emit light rays with corresponding wavelengths through adjusting the light sources, thereby obtaining images of objects with enough quality, and further accurately obtaining the surface information of the objects. As still another example, each of the 5 light sources may emit light at wavelengths of 420nm, 440nm, 460nm, 480nm, 510nm, 530nm, 550nm, 600nm, 640nm, 680nm, etc.

Further, before shooting the reference object by the local camera of the execution device, the user may issue an adjustment request to the local execution device, so that the local execution device performs parameter adjustment on the camera, where the parameter adjustment operation includes at least one of the following:

(1) The exposure time of the camera is adjusted so that the exposure time of the camera is smaller than or equal to a preset first threshold, wherein the preset first threshold can be understood as an exposure time threshold required by a business environment in which an object is located, and the size of the exposure time can be set according to actual requirements. For example, let the local execution device be an industrial camera and the reference object be a workpiece. In a production test environment with a production efficiency of 100pcs/min, which requires a total processing time of the workpiece to be 600ms or less, assuming that a time required for a subsequent process of an image of the workpiece (for example, a process of acquiring surface information of the workpiece based on the image of the workpiece) after the industrial camera photographs the industry to obtain the image of the workpiece is 500ms (depending on a resolution of the image of the workpiece), it is necessary to adjust an exposure time of a camera of the industrial camera to be 100ms or less to satisfy a time requirement of the environment on the workpiece. It should be understood that this example is only schematically illustrated with an exposure time threshold of 100ms, and is not limiting on the size of the exposure time threshold in the present application.

(2) The brightness of the plurality of light sources in the camera is adjusted so that the brightness of the plurality of light sources is greater than or equal to a preset second threshold, wherein the preset second threshold can be understood as a light source brightness threshold required by the imaging effect of the object with enough quality, and the size of the preset second threshold can be set according to actual requirements. For example, the brightness of the light source is continuously adjusted, so that the camera obtains the image of the reference object, and the pixel value of each pixel point in the area where the reference object is located is 70% -80% of the maximum pixel value (255), that is, the pixel value of each pixel point is between 178 and 204, so that the imaging effect of the reference object is better. It should be understood that this example is only schematically illustrated with respect to the range of 178 to 204 pixel values of each pixel in the region where the reference object is located, and is not limited to the size of the pixel values of each pixel in the region where the reference object is located in the present application, that is, is not limited to the size of the light source brightness threshold.

(3) The wavelengths of light rays emitted by a plurality of light sources in the camera are adjusted so that the signal to noise ratio of an image of a reference object shot by the camera is greater than or equal to a preset third threshold, wherein the preset third threshold can be understood as an image signal to noise ratio threshold required by the imaging effect of the object with enough quality, and the size of the threshold can be set according to actual requirements. For example, after determining the exposure time of the camera, the camera may be covered by the lens cover, the light source of the camera is kept off, no light enters the sensor of the camera, then an image is captured by using the camera in this state, and the pixel value of each pixel point in the image should be theoretically 0, but in practice, due to the dark current and the existence of shot noise, the pixel value of each pixel point in the noise image captured by the camera in this state is usually between 0 and 3, and then the average noise level of the noise image may be calculated by the following formula:

In the above, g _noise (x, y) is the pixel value of the pixel at the (x, y) position in the noise image, w is the width of the noise image, h is the height of the noise image, and w×h is the number of pixels of the noise image (i.e., the resolution of the image).

Then, a lens cover on the camera is opened, and a light source in the camera is started, so that the light source sends light with a certain wavelength to a reference object to obtain an image of the reference object, and the average signal level of the image of the reference object is as follows:

in the above, g _noise (x, y) is the pixel value of the pixel at the (x, y) position in the image of the reference object, and w×h is the number of pixels of the reference image (i.e., the resolution of the image).

Then, based on the average noise level of the noise image and the average signal level of the image of the reference object, the signal-to-noise ratio can be calculated:

ratio＝Signal/Noise (4)

if the signal-to-noise ratio is smaller than the preset third threshold value, the wavelength of the light emitted by the light source is unsuitable, the wavelength of the light emitted by the light source can be readjusted, the signal-to-noise ratio is calculated again until the signal-to-noise ratio is larger than or equal to the preset third threshold value, the wavelength of the light emitted by the light source is suitable, and the imaging effect of the reference object can be better.

(4) The definition of the image of the reference object shot by the camera is adjusted so that the definition of the image of the reference object shot by the camera is larger than or equal to a preset fourth threshold, wherein the preset fourth threshold can be understood as a definition threshold required by the imaging effect of the object with enough quality, and the size of the definition threshold can be set according to actual requirements. For example, the sharpness of the image of the reference object may be evaluated here in a Brenner gradient function by a gradient method, a gray variance method, an entropy function, or the like, which function is shown in the following formula:

In the above formula, D (g) is a sharpness evaluation value of an image. In actual operation, the adjustment of definition can be realized by rotating the focusing ring of the camera, and when the definition evaluation value of the reference image is larger than or equal to the preset fourth threshold value, the image is clear and available at the moment, and the imaging effect is better, so that the adjustment of definition can be finished.

It should be noted that, of the four operations of adjusting the camera parameters, one or more operations may be selected for execution. If the four operations need to be executed in the actual application, the four operations may be executed sequentially in a certain order, where the flow is shown in fig. 6 (fig. 6 is a schematic flow diagram of adjusting the camera according to the embodiment of the present application): after the camera is adjusted, the exposure time of the camera can be adjusted first, whether the exposure time of the camera is smaller than or equal to a preset first threshold value is judged, if not, the exposure time of the camera is adjusted again, if yes, the brightness of a light source in the camera is adjusted, then the brightness of the light source is judged to be larger than or equal to a preset second threshold value, if not, the brightness of the light source is adjusted again, if yes, the wavelength of light rays sent by the light source in the camera is adjusted, then whether the signal-to-noise ratio of an image shot by the camera is larger than or equal to a preset third threshold value is judged, if not, the wavelength of the light rays sent by the light source in the camera is adjusted again, if yes, the definition of the image shot by the camera is adjusted, then the definition of the image shot by the camera is judged to be larger than or equal to a preset fourth threshold value, if not, then the definition of the image shot by the camera is adjusted again, and finally the adjustment is finished.

After the parameters of the camera are adjusted, the local execution equipment can shoot the reference object through the adjusted camera to obtain an image of the reference object. For example, let the camera of the industrial camera include M light sources (M is greater than or equal to 3), and K reference objects (K is greater than or equal to 1) exist. For any one of the K reference objects, the camera may sequentially turn on the light sources at different positions according to a preset sequence (for example, turn on the M light sources according to the sequence of the M light sources or the remaining custom sequence), and shoot the reference object, so that M images of the reference object can be obtained and considered as a set of images of the reference object. Similar operations can be performed for the rest of the reference objects other than the reference object, so that K sets of images of the reference object (one set of images of the reference object contains M images of a certain reference object) can be obtained for a total of k×m images.

402. The local execution device transmits an image of the reference object to the remote execution device.

After obtaining the image of the reference object, the local execution device may send the image of the reference object to the remote execution device. Still as in the example above, after obtaining the images of the K sets of reference objects, the industrial camera may send the images of the K sets of reference objects to the cloud server.

403. The remote execution device acquires surface information of the reference object based on the image of the reference object, the surface information of the reference object including a reflectivity of the reference object and/or a gaussian curvature of the reference object.

After obtaining the image of the reference object, the remote execution device may calculate the image of the reference object, so as to obtain surface information (stereo information) of the reference object, where the surface information of the reference object includes the following three cases: (1) Reflectivity of the reference object (which may also be referred to as a reflectivity map of the reference object); (2) The gaussian curvature of the reference object (which may also be referred to as the gaussian curvature map of the reflecting object); (3) Reflectivity of the reference object and gaussian curvature of the reference object.

In particular, the remote execution device may acquire the surface information of the reference object by:

the remote execution device can calculate the image of the reference object based on the photometric stereo algorithm, so as to obtain the surface information of the reference object. Still as in the above example, based on the photometric stereo algorithm, for a certain reference object of the K reference objects, the following relation exists:

in the above formula, I is the relative intensity of the image of the reference object, Is the unit direction vector of the light source, < >>For the normal vector of the reference object surface, ρ is the reflectivity of (all points in the surface of) the reference object. Wherein, for any point in the surface of the reference object, the normal vector of that point +.>The following formula can be used:

where x, y and z are the positions of the point in the surface of the reference object (i.e., the abscissa, ordinate and ordinate of the point in the surface of the reference object), z=f (x, y), and for convenience of description, the point may be hereinafter referred to as a point at the (x, y) position in the surface of the reference object. In general, a point at the (x, y) position in the surface of the reference object may correspond to a pixel point at the (x, y) position in the image of the reference object.

In equation (7), p (x, y) and q (x, y) are unknowns, and the relative intensity I of the image of the reference object and the unit direction vector of the light sourceIs a known quantity, in order to solve for the reflectivity ρ (x, y) of points at (x, y) positions in the surface of the reference object, and p (x, y), it may be assumed that m=3 light sources are present in the camera of the local execution device, so that the remote execution device may obtain an image obtained by photographing the reference object based on these 3 light sources, and the remote execution device may perform the following solving process based on the 3 images of the reference object:

For points at (x, y) locations in the surface of the reference object, the following system of photometric stereo equations can be constructed:

in the above, I ₁ (x, y) is the pixel point at the (x, y) position in the 1 st image of the reference object (i.e. the camera starts the 1 st light source to shoot the reference object), I ₂ (x, y) is the pixel point at the (x, y) position in the 2 nd image of the reference object (i.e. the camera turns on the 2 nd light source, the image obtained by shooting the reference object), I ₃ (x, y) is the pixel point at the (x, y) position in the 3 rd image of the reference object (namely, the 3 rd light source is started by the camera to shoot the reference object),is the unit direction vector of the 1 st light source, < >>For the unit direction vector of the 2 nd light source, is->Is the unit direction vector of the 3 rd light source.

Due to the normal vector of a point in the (x, y) position in the surface of the reference objectCan be expressed as n= [ n ] ₁ ,n ₂ ,n ₃ ] ^T ，/>Can be expressed as N ₁ ＝[n ₁₁ ，n ₁₂ ，n ₁₃ ] ^T ，/>Can be expressed as N ₂ ＝[n ₂₁ ，n ₂₂ ，n ₂₃ ] ^T ，/>Can be expressed as N ₃ ＝[n ₃₁ ，n ₃₂ ，n ₃₃ ] ^T Therefore, the following steps are adopted:

and let I "= [ I ] ₁ (x,y),I ₂ (x,y),I ₃ (x,y)] ^T The following steps are:

I``＝ρ(x,y)Nn (10)

based on equation (10), the reflectivity ρ (x, y) of a point at the (x, y) position in the surface of the reference object can be solved:

ρ(x,y)＝|N ^-1 I``| (11)

accordingly, the normal vector n of a point at the (x, y) position in the surface of the reference object can be solved:

n＝(1/ρ)N ^-1 I`` (12)

It can be seen that the normal vector n of the point at the (x, y) position in the surface of the reference object is determined, i.e. p (x, y) and q (x, y) as described above are determined, so that a second order derivative can be made of p (x, y) and q (x, y), i.e.:

then, the gaussian curvature of a point at the (x, y) position in the surface of the reference object can be solved by the following formula:

due to p _x 、p _y 、q _x Q _y All are unknowns, and these four quantities can be solved by means of gaussian derivative filtering, so that the gaussian curvature K (x, y) of a point at the (x, y) position in the surface of the reference object can be found.

Likewise, for points at the rest of the surface of the reference object, the remote execution device may also perform the same operations as for points at the (x, y) position, so that the remote execution device may obtain the reflectivity ρ of (all points in the surface of) the reference object, as well as the gaussian curvature K of (all points in the surface of) the reference object.

Likewise, for the rest of the K reference objects except for the reference object, the remote execution device may also perform the same operation as performed on the reference object, so that the remote execution device may obtain the reflectivities of the K reference objects and the gaussian curvatures of the K reference objects, that is, the surface information of the K reference objects.

404. And the remote execution equipment processes the surface information of the reference object through the feature extraction model to obtain the features of the reference object.

After obtaining the surface information of the reference object, the remote execution device may input the surface information of the reference object to a feature extraction model (trained neural network model) to process (e.g., perform convolution processing, normalization processing, etc.) the surface information of the reference object through the feature extraction model, thereby obtaining the features of the reference object.

Further, the remote execution device may further perform calculation on the features of the reference object to obtain statistics of the features of the reference object, that is, perform a first calculation on the features of the reference object (i.e., calculate a mean vector), thereby obtaining a mean vector of the features of the reference object, and perform a second calculation on the features of the reference object (i.e., calculate a covariance matrix), thereby obtaining a covariance matrix of the features of the reference object.

Still as in the above example, after obtaining the reflectivities of the K reference objects and the gaussian curvatures of the K reference objects, the remote execution device may perform data augmentation on the reflectivities of the K reference objects and the gaussian curvatures of the K reference objects, so as to obtain X reflectivities and X gaussian curvatures (X is greater than or equal to 2K), that is, X pair reflectivities and gaussian curvatures, and it is understood that, among the K reference objects, one reference object corresponds to at least one pair of X pair reflectivities and gaussian curvatures. For any one of the X pairs of reflectivity and gaussian curvature, the remote execution device may input the pair of reflectivity and gaussian curvature to the feature extraction model, such that the feature extraction model processes the pair of reflectivity and gaussian curvature to yield Z features (Z is greater than or equal to 1), which may be considered a set of features. For the remaining pairs of reflectivity and gaussian curvature, the same operations as performed for the pairs of reflectivity and gaussian curvature can also be performed, so that X sets of features (one set of features comprising Z features corresponding to a pair of reflectivity and gaussian curvature) can be obtained. In the X sets of features, a first calculation may be performed on a 1 st feature of each set of features to obtain a mean vector of the 1 st feature of the K reference objects, a first calculation may be performed on a 2 nd feature of each set of features to obtain a mean vector of the 2 nd feature of the K reference objects, and a first calculation may be performed on a Z-th feature of each set of features to obtain a mean vector of the Z-th feature of the K reference objects. Similarly, a second calculation may also be performed on the 1 st feature of each set of features to obtain a covariance matrix of the 1 st feature of the K reference objects, a second calculation may be performed on the 2 nd feature of each set of features to obtain a covariance matrix of the 2 nd feature of the K reference objects, and a second calculation may be performed on the Z-th feature of each set of features to obtain a covariance matrix of the Z-th feature of the K reference objects.

405. The remote execution device sends information indicating the characteristics of the reference object to the local execution device.

After obtaining statistics of the features of the reference object (including a mean vector of the features of the reference object and a covariance matrix of the features of the reference object), the remote execution device may send the statistics of the features of the reference object to the local execution device as information for indicating the features of the reference object, so that the subsequent local execution device may implement surface evaluation of the target object.

Still as in the above example, after obtaining the mean vector of the 1 st feature of the K reference objects, the mean vector of the 2 nd feature of the K reference objects, the term, the mean vector of the Z-th feature of the K reference objects, the covariance matrix of the 1 st feature of the K reference objects, the covariance matrix of the 2 nd feature of the K reference objects, the term, the covariance matrix of the Z-th feature of the K reference objects, the remote execution device may send these information to the local execution device.

406. The local execution device acquires an image of the target object.

After obtaining the information indicating the characteristics of the reference object, i.e. after obtaining statistics of the characteristics of the reference object, the local execution device may determine the characteristics of the reference object based on the information and store the information. The local execution device has the function of evaluating the surface of the object. Then, when the user needs to evaluate whether the surface of the target object is qualified, that is, whether the surface of the target object has a defect is detected, the image of the target object can be acquired through the local execution device.

Specifically, since the local execution device is deployed with a camera, the local execution device can shoot the target object through the camera, so that an image of the target object is acquired. It should be noted that, for the description of the local camera of the execution device, reference may be made to the related description portion of the local camera of the execution device in step 401, which is not repeated herein.

Further, before the target object is photographed by the local camera of the execution device, the user may first perform parameter adjustment on the local camera of the execution device, and description about the parameter adjustment operation may refer to step 401, where relevant description portions of the parameter adjustment performed by the user on the local camera of the execution device before the reference object is photographed by the local camera of the execution device are not repeated herein.

After the parameters of the camera are adjusted, the local execution equipment can shoot the target object through the adjusted camera to obtain an image of the target object. Still as the above example, the camera of the industrial camera includes M light sources, and the user needs to evaluate the surface of a certain object, i.e. the target object, so that the camera of the industrial camera can be controlled to sequentially turn on the light sources at different positions to shoot the target object, so that M images of the target object can be obtained.

407. The local execution device obtains surface information of the target object based on an image of the target object, the surface information of the target object comprising a reflectivity of the target object and/or a gaussian curvature of the target object.

After obtaining the image of the target object, the local execution device may calculate the image of the target object, so as to obtain surface information (stereo information) of the target object, where the surface information of the target object includes the following three conditions: (1) The reflectivity of the target object (which may also be referred to as a reflectivity map of the target object); (2) The gaussian curvature of the target object (which may also be referred to as the gaussian curvature map of the reflecting object); (3) The reflectivity of the target object and the gaussian curvature of the target object.

Specifically, the local execution device may acquire the surface information of the target object by:

the local execution device can calculate the image of the target object based on the photometric stereo algorithm, so as to obtain the surface information of the target object. It should be noted that, regarding the description that the local execution device obtains the surface information of the target object by using the photometric stereo algorithm, reference may be made to the description that the remote execution device obtains the surface information of the reference object by using the photometric stereo algorithm in step 403, which is not described herein.

408. The local execution equipment processes the surface information of the target object through the feature extraction model to obtain the features of the target object.

After obtaining the surface information of the target object, the local execution device may input the surface information of the target object to a feature extraction model (the feature extraction model of the local execution device and the feature extraction model of the remote execution device are the same neural network model), so as to process (for example, perform convolution processing, normalization processing, etc.) the surface information of the target object through the feature extraction model, thereby obtaining the features of the target object.

Further, the local execution device may further perform calculation on the features of the target object to obtain statistics of the features of the target object, that is, perform first calculation on the features of the target object (i.e., calculate a solution to a mean vector), thereby obtaining a mean vector of the features of the target object, and perform second calculation on the features of the target object (i.e., calculate a solution to a covariance matrix), thereby obtaining a covariance matrix of the features of the target object.

Still as in the example above, after obtaining the reflectivity of the target object and the gaussian curvature of the target object, the local execution device may perform data augmentation on the reflectivity of the target object and the gaussian curvature of the target object, so as to obtain Y reflectivities and Y gaussian curvatures (Y is greater than or equal to 2), that is, Y pair reflectivities and gaussian curvatures, and it will be understood that these Y pair reflectivities and gaussian curvatures correspond to the target object. For any one of the Y pairs of reflectivity and gaussian curvature, the local execution device may input the pair of reflectivity and gaussian curvature to the feature extraction model, such that the feature extraction model processes the pair of reflectivity and gaussian curvature to yield Z features (Z is greater than or equal to 1), which may be considered a set of features. For the remaining pairs of reflectivity and gaussian curvature, the same operations as performed for the pairs of reflectivity and gaussian curvature can also be performed, so that Y sets of features (one set of features comprising Z features corresponding to a pair of reflectivity and gaussian curvature) can be obtained. Among the Y sets of features, a first calculation may be performed on a 1 st feature of each set of features to obtain a mean vector of the 1 st feature of the target object, a first calculation may be performed on a 2 nd feature of each set of features to obtain a mean vector of the 2 nd feature of the target object, and a first calculation may be performed on a Z-th feature of each set of features to obtain a mean vector of the Z-th feature of the target object. Similarly, a second calculation may be performed on the 1 st feature of each set of features to obtain a covariance matrix of the 1 st feature of the target object, a second calculation may be performed on the 2 nd feature of each set of features to obtain a covariance matrix of the 2 nd feature of the target object.

409. The local execution device determines that the target object is a surface-qualified object or a surface-disqualified object based on the characteristics of the target object and the characteristics of the reference object.

After determining the characteristics of the target object and the characteristics of the reference object, the local execution device can compare the characteristics of the target object with the characteristics of the reference object, so as to judge whether the surface of the target object is qualified, i.e. whether the surface of the target object has defects.

Specifically, the local execution device may determine whether the surface of the target object is acceptable by:

(1) After obtaining the statistics of the features of the reference object and the statistics of the features of the target object, the local execution device may calculate the statistics of the features of the target object and the statistics of the features of the reference object, thereby obtaining a score of the target object, where the score may be used to evaluate the surface of the target object.

Notably, the local execution device may calculate the score of the target object by: the local execution device may first calculate a difference between the mean vector of the feature of the target object and the mean vector of the feature of the reference object, then fuse the covariance matrix of the feature of the target object with the covariance matrix of the feature of the reference object, and then calculate a distance (e.g., mahalanobis distance (mahalanobis distance) between the difference and the fused matrix, etc.), where the distance may be used as a score of the target object.

Still further, as in the above example, the local execution device may first calculate a difference between the mean vector of the 1 st feature of the K reference objects and the mean vector of the 1 st feature of the target object, fuse the covariance matrix of the 1 st feature of the K reference objects with the covariance matrix of the 1 st feature of the target object, and then calculate a mahalanobis distance between the difference and the fused matrix. Then, the local execution device may calculate a difference between the mean vector of the 2 nd feature of the K reference objects and the mean vector of the 2 nd feature of the target object, fuse the covariance matrix of the 2 nd feature of the K reference objects with the covariance matrix of the 2 nd feature of the target object, and calculate a mahalanobis distance between the difference and the fused matrix. And the like, until the local execution equipment calculates the difference value between the mean vector of the Z-th feature of the K reference objects and the mean vector of the Z-th feature of the target object, fuses the covariance matrix of the Z-th feature of the K reference objects and the covariance matrix of the Z-th feature of the target object, and then calculates the Markov distance between the difference value and the matrix obtained by fusion. In this way, the local execution device may obtain Z mahalanobis distances, where the Z mahalanobis distances may be regarded as Z scores, and the Z scores may form a score map, and since the size of the score map is not equal to the size of the image of the target object (because feature extraction is performed on the surface information of the target object obtained based on the image of the target object), the size of the feature of the target object is often not equal to the size of the image of the target object, and it may be seen that the size of the score map obtained based on the feature of the target object is not equal to the size of the image of the target object), so that the image scaling operation may be performed on the score map, so that the size of the processed score map is equal to the size of the image of the target object, that is, the processed score map includes w×h scores, where the w×h scores are in one-to-one correspondence with w×h pixels in the target image, that is, the score at the (x, y) position in the processed score map is the score of the point at the (x, y) position in the surface of the target object. The processed score map may then be regarded as a score of the target object.

(2) After the score of the target object is obtained, the local execution device can detect whether the score of the target object is larger than a preset fifth threshold value, and the size of the preset fifth threshold value can be set according to actual requirements, and the method is not limited herein.

(3) If the score of the target object is greater than the preset fifth threshold value, indicating that the surface of the target object has a defect, the local execution equipment can determine that the target object is an object with unqualified surface. It should be noted that, the points in the surface of the target object are regarded as a plurality of (surface) areas of the target object, so that the score of the target object also includes the scores of the plurality of areas of the target object, and if the score of at least one area of the target object is greater than the preset fifth threshold, the local execution device may determine that the surface of the target object is an unqualified object. Still as in the above example, in the processed score map, if there is a score of a part or all of the scores being greater than the preset fifth threshold, the local execution device may determine that the target object is an object with a surface failure, that is, the surface of the target object has a defect.

Further, after determining that the target object is an object whose surface is not acceptable, the local execution device may determine which partial region of the target object is an area whose surface is not acceptable based on the scores of the plurality of regions of the target object, that is, the local execution device may determine, among the plurality of regions of the target object, the region whose score is greater than the preset fifth threshold as the area whose surface is not acceptable in the target object. Still as in the above example, based on the processed score map, the local execution device may determine which points on the surface of the target object have scores greater than a preset fifth threshold, so that the points on these positions may be determined as areas of the surface failure in the target object.

(4) If the score of the target object is smaller than or equal to a preset fifth threshold value, the target object is indicated to have no defect on the surface, and the local execution equipment can determine that the target object is an object with qualified surface. That is, if the score of all the areas of the target object is less than or equal to the preset fifth threshold, the local execution device may determine that the target object is a surface-qualified object. Still as in the above example, in the processed score map, if all scores are less than or equal to the preset fifth threshold, the local execution device may determine that the target object is a surface-qualified object, that is, that the surface of the target object has no defect.

Further, after determining that the target object is a surface-qualified object, the local execution device may further send a reference update request to the remote execution device, so that the remote execution device determines the target object indicated by the reference update request as a new reference object. It should be noted that the reference update request may include an image of the target object, surface information of the target object, characteristics of the target object, and statistics of the characteristics of the target object, and the remote execution device may determine the target object as a new reference object after receiving the request, and store the content included in the request. The remote execution device may then update the statistics of the features of the new reference object to the statistics of the features of the original reference object (e.g., add the statistics of the features of the new reference object to the statistics of the features of the original reference object, replace some or all of the statistics of the features of the original reference object with the statistics of the features of the new reference object, etc.), resulting in updated statistics. Finally, the remote execution device may send the updated statistics to the local execution device, such that the local execution device replaces the statistics of the features of the original reference object with the updated statistics. When a user needs to make a surface assessment of a new target object, the local execution device may determine whether the surface of the new target object is acceptable based on the updated statistics and statistics of the characteristics of the new target object.

It should be understood that in the present embodiment, the number of feature extraction models is only schematically described as an example, and the number of feature extraction models in the present application is not limited. In practical application, the number of the feature extraction models can be more than one, so that the features of the reference object obtained by one feature extraction model can be regarded as a feature set of the reference object, and after the far-end execution device inputs the surface information of the reference object into the feature extraction models, the feature sets of the reference objects can be obtained correspondingly.

Similarly, the features of the target object obtained by one feature extraction model may be regarded as a feature set of the target object, and the local execution device may obtain the feature sets of the plurality of target objects after inputting the surface information of the target object to the plurality of feature extraction models. Since statistics of the characteristics of a set of target objects can be obtained based on a set of characteristics of a target object, the remote execution device can obtain statistics of the characteristics of multiple sets of target objects. In this way, the local execution device can obtain statistics of the characteristics of multiple groups of reference objects and statistics of the characteristics of multiple groups of target objects, and the statistics of the characteristics of the multiple groups of reference objects and the statistics of the characteristics of the multiple groups of target objects are in one-to-one correspondence.

Then, for any set of statistics of the characteristics of the target object, the local execution device may calculate the statistics of the characteristics of the target object and the statistics of the characteristics of its corresponding reference object, thereby obtaining a set of scores of the target object (scores of multiple regions containing the target object). Similar operations can be performed by local execution devices for statistics of features of the remaining target objects, so that scores of multiple groups of target objects can be obtained. After obtaining the scores of the multiple groups of target objects, the local execution device may fuse the scores of the multiple groups of target objects (for example, weight and sum the scores of the multiple groups of target objects to obtain a final score of the target object), then determine whether the final score of the target object (including the final scores of multiple regions of the target object) is greater than a preset fifth threshold value, so as to determine whether the surface of the new target object is qualified, that is, if the final score of at least one region of the target object is greater than the preset fifth threshold value, the local execution device may determine that the target object is an object with a surface unqualified, and if the final score of all regions of the target object is less than or equal to the preset fifth threshold value, the local execution device may determine that the target object is an object with a surface qualified.

Still as in the above example, provided that there are J feature extraction models (J is greater than 1), the remote execution device obtains X pairs of reflectivities and gaussian curvatures, which can be input into the J feature extraction models, respectively. For any one of the J feature extraction models, the feature extraction model may process the X pair reflectivity and gaussian curvature to obtain X sets of features, which are considered as a feature set. The remote execution device may then obtain, based on the feature set, a mean vector of the 1 st feature of the K reference objects, a mean vector of the 2 nd feature of the K reference objects, a mean vector of the Z-th feature of the K reference objects, a covariance matrix of the 1 st feature of the K reference objects, a covariance matrix of the 2 nd feature of the K reference objects, a covariance matrix of the Z-th feature of the K reference objects, the Z mean vectors and the Z covariance matrices may be considered statistics of the features of the set of reference objects. Similar operations can be performed by the remote execution device for the rest of the J feature extraction models, so statistics of the features of the J groups of reference objects can be obtained and issued to the local execution device.

Similarly, after the local execution device obtains the Y-pair reflectivity and the gaussian curvature, the Y-pair reflectivity and the gaussian curvature may be input into the J feature extraction models, respectively. For any one of the J feature extraction models, the feature extraction model may process the Y pair reflectivity and gaussian curvature to obtain Y sets of features, which are considered as a feature set. The local execution device may then obtain, based on the feature set, a mean vector of the 1 st feature of the target object, a mean vector of the 2 nd feature of the target object, a mean vector of the Z-th feature of the target object, a covariance matrix of the 1 st feature of the target object, a covariance matrix of the 2 nd feature of the target object, a covariance matrix of the Z-th feature of the target object, the Z mean vectors and the Z covariance matrices being considered statistics of a set of features of the target object. For the rest of the J feature extraction models, the local execution device can also execute similar operation, so that statistics of the features of the J groups of target objects can be obtained.

Because the statistics of the features of the J-group reference object and the statistics of the features of the J-group target object are in one-to-one correspondence, the local execution device may calculate the statistics of the features of the 1 st group reference object and the statistics of the features of the 1 st group target object to obtain a 1 st score map, calculate the statistics of the features of the 2 nd group reference object and the statistics of the features of the 2 nd group target object to obtain a 2 nd score map, and calculate the statistics of the features of the J-group reference object and the statistics of the features of the J-group target object to obtain a J-th score map, so that the local execution device may obtain J score maps (i.e., scores of the plurality of groups of target objects), and because the J score maps are generally different in size, perform an image scaling operation on the J score maps so that the J score maps are the same as the image size of the target object, and perform weighted summation on the J-processed score maps to obtain a final score map (i.e., a final score of the target object).

Because the final scoring graph contains the final scores of all points on the surface of the target object, if part of the scores or all the scores in the graph are greater than a preset fifth threshold, the local execution equipment can determine that the target object is an object with a disqualified surface, and if all the scores in the graph are less than or equal to the preset fifth threshold, the local execution equipment can determine that the target object is an object with a qualified surface.

It should also be appreciated that in the above example, the value of Z may be the same or different for different feature extraction models among the J feature extraction models.

It should be further understood that, in this embodiment, the object surface evaluation method provided in the embodiment of the present application is only schematically described by using a local execution device and a remote execution device together, and in practical application, the object surface evaluation method provided in the embodiment of the present application may also be completed by the local execution device alone, that is, in steps 401 to 409, all the steps completed by the remote execution device may be completed by the local execution device, for example, the local execution device obtains statistics of features of the reference object based on an image of the reference object, the local execution device determines the target object as a new reference object, updates the statistics of features of the original reference object, and so on.

In the embodiment of the present application, after the image of the target object is acquired, the local execution device may acquire surface information of the target object based on the image of the target object, where the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

Further, the camera of the local execution device comprises a plurality of light sources, each light source in the plurality of light sources is a multi-wavelength light source, and before the camera shoots the object with the surface of various colors, the wavelength of the light rays emitted by the plurality of light sources can be adjusted to adapt to the object with the surface of various colors, so that the imaging effect of the object with the surface of various colors is high enough, that is, the image of the object shot by the camera has good quality, and the method is favorable for extracting the surface information of the object with sufficient accuracy.

Furthermore, the evaluation mode provided by the embodiment of the application can be applied to a system comprising local execution equipment and remote execution equipment, the number of hardware equipment required by the system is often less, the hardware cost and the customization difficulty required by the system deployment are reduced, the system can be opened for various users, and the system does not need to put forward high professional requirements on the users, so that the system can be widely applied to various business scenes.

The foregoing is a detailed description of the object surface evaluation method provided by the embodiment of the present application, and the model training method provided by the embodiment of the present application will be described below, and fig. 7 is a schematic flow chart of the model training method provided by the embodiment of the present application, as shown in fig. 7, where the method includes:

701. An image of an object to be processed is acquired.

In this embodiment, when a model to be trained (neural network model to be trained) is required to be trained, a batch of training data may be acquired first, where the batch of training data includes images of objects to be processed, where the objects to be processed may be objects with qualified surfaces or objects with unqualified surfaces. It should be noted that the actual characteristics of the object to be processed are known.

702. The method comprises the steps of obtaining surface information of an object to be processed based on an image of the object to be processed, wherein the surface information of the object to be processed comprises reflectivity of the object to be processed and/or Gaussian curvature of the object to be processed.

After the image of the object to be processed is obtained, the image of the object to be processed can be calculated, so that the surface information of the object to be processed is obtained, wherein the surface information of the object to be processed comprises the reflectivity of the object to be processed and/or the Gaussian curvature of the object to be processed.

In one possible implementation, the image of the object to be processed may be calculated based on a photometric stereo algorithm, so as to obtain the surface information of the object to be processed.

It should be noted that, for the description of step 702, reference may be made to the related description of step 403 or step 407 in the embodiment shown in fig. 4, which is not repeated herein.

703. And processing the surface information of the object to be processed through the model to be trained to obtain the characteristics of the object to be processed.

After the surface information of the object to be processed is obtained, the surface information of the object to be processed may be input to the feature extraction model, so that the surface information of the object to be processed is processed (for example, convolution processing, normalization processing, etc.) through the model to be trained, thereby obtaining the features of the object to be processed, that is, the predicted features of the object to be processed.

It should be noted that, with respect to the description of step 703, the description of step 404 or step 408 in the embodiment shown in fig. 4 is omitted here.

704. Based on the characteristics of the object to be processed and the real characteristics of the object to be processed, a target loss is obtained, the target loss being used to indicate a difference between the characteristics of the object to be processed and the real characteristics of the object to be processed.

After the predicted features of the object to be processed are obtained, the predicted features of the object to be processed and the real features of the object to be processed can be calculated through a preset target loss function, so that target losses are obtained, and the target losses are used for indicating differences between the predicted features of the object to be processed and the real features of the object to be processed.

705. And updating parameters of the model to be trained based on the target loss until the model training condition is met, and obtaining the feature extraction model.

After obtaining the target loss, the parameters of the model to be trained may be updated based on the target loss, and the model to be trained after updating the parameters may be continuously trained (i.e., steps 701 to 705 may be re-performed) by using the next batch of training data until the model training condition (e.g., the target loss converges, etc.) is met, to obtain the feature extraction model in the embodiment shown in fig. 4.

The feature extraction model obtained by the embodiment of the application can realize the function of extracting the features of the target object in the process of carrying out surface recognition on the target object. Specifically, after acquiring the image of the target object, the local execution device may acquire surface information of the target object based on the image of the target object, wherein the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object. The local execution device may then input the surface information of the target object into a feature extraction model, thereby obtaining features of the target object. Finally, the local execution device may determine that the target object is a surface-eligible object or a surface-ineligible object based on the characteristics of the target object and the characteristics of the reference object, wherein the reference object is a surface-eligible object. In the foregoing process, no matter what type of product the target object is, the local execution device can utilize the feature extraction model to process the surface information of the target object so as to automatically extract the features of the target object, and compare the features of the target object with the features of the reference object with qualified surfaces, so as to determine whether the surfaces of the target object are qualified, and the new evaluation mode can extract the features of various objects through the neural network model, so that the artificial factors are avoided, and the unified reference standard for evaluating the surfaces of various objects is introduced, so that the considered factors are more comprehensive, the surface evaluation of the objects is realized based on the new mode, and the higher accuracy can be realized.

The foregoing is a detailed description of the model training method provided by the embodiment of the present application, and the object surface evaluation device and the model training device provided by the embodiment of the present application will be described below. Fig. 8 is a schematic structural diagram of an object surface evaluation device according to an embodiment of the present application, as shown in fig. 8, where the device may be disposed in a processor of a local execution device, and the processor is connected to a camera of the local execution device, where the camera includes a plurality of light sources located at different positions, and each of the plurality of light sources may emit light with a plurality of wavelengths, and the device includes:

a first obtaining module 801, configured to control the camera to start a plurality of light sources according to a preset sequence, so as to shoot a target object, and obtain an image of the target object;

a second acquisition module 802, configured to acquire surface information of a target object based on an image of the target object, where the surface information of the target object includes a reflectivity of the target object and/or a gaussian curvature of the target object;

the processing module 803 is configured to process surface information of the target object through the feature extraction model, so as to obtain features of the target object;

the first determining module 804 is configured to determine that the target object is a surface-qualified object or a surface-unqualified object based on the characteristics of the target object and the characteristics of the reference object, where the reference object is a surface-qualified object.

In one possible implementation, the second obtaining module 802 is configured to calculate an image of the target object based on a photometric stereo algorithm, so as to obtain surface information of the target object.

In one possible implementation, the first determining module 804 is configured to: receiving statistics of features of the reference, the statistics of features of the reference comprising mean vectors of features of the reference and covariance matrices of features of the reference; calculating the characteristics of the target object to obtain statistics of the characteristics of the target object, wherein the statistics of the characteristics of the target object comprise mean vectors of the characteristics of the target object and covariance matrixes of the characteristics of the target object; calculating statistics of the characteristics of the target object and statistics of the characteristics of the reference object to obtain scores of the target object; if the score of the target object is larger than a preset fifth threshold value, determining that the target object is an object with unqualified surface; and if the score of the target object is smaller than or equal to a preset fifth threshold value, determining that the target object is the object with qualified surface.

In one possible implementation, the first determining module 804 is configured to: controlling the camera to sequentially start a plurality of light sources so as to shoot a reference object and obtain an image of the reference object; acquiring surface information of a reference object based on an image of the reference object, the surface information of the reference object including reflectivity of the reference object and/or gaussian curvature of the reference object; processing the surface information of the reference object through the feature extraction model to obtain the features of the reference object; and calculating the features of the reference object to obtain statistics of the features of the reference object, wherein the statistics of the features of the reference object comprise mean vectors of the features of the reference object and covariance matrixes of the features of the reference object.

In one possible implementation, the apparatus further includes: the third acquisition module is used for shooting the reference object through the camera to obtain an image of the reference object; and the second transmitting module is used for transmitting the image of the reference object.

Fig. 9 is a schematic structural diagram of a statistic acquiring device according to an embodiment of the present application, where, as shown in fig. 9, the device is deployed in a remote execution device, and the device includes:

a first acquisition module 901, configured to acquire an image of a reference object;

a second obtaining module 902, configured to obtain surface information of a reference object based on an image of the reference object, where the surface information of the reference object includes a reflectivity of the reference object and/or a gaussian curvature of the reference object;

the processing module 903 is configured to process the surface information of the reference object through the feature extraction model, so as to obtain features of the reference object;

A calculation module 904, configured to calculate a feature of the reference object, to obtain a statistic of the feature of the reference object, where the statistic of the feature of the reference object includes a mean vector of the feature of the reference object and a covariance matrix of the feature of the reference object;

a transmitting module 905, configured to transmit statistics of features of the reference object.

In the embodiment of the application, the remote execution equipment can provide a unified reference standard (namely, statistics of the characteristics of the reference object) for carrying out surface recognition on various objects for the local execution equipment so as to jointly realize the surface recognition on the objects by combining the local execution equipment, so that the quantity of hardware equipment required by a system formed by the remote execution equipment and the local execution equipment is often less, the hardware cost and the customization difficulty required by the system deployment are reduced, the remote execution equipment can be opened for various users, and the remote execution equipment does not need to put forward excessively high professional requirements on the users, so that the remote execution equipment can be widely applied to various business scenes.

Fig. 10 is a schematic structural diagram of a model training apparatus according to an embodiment of the present application, as shown in fig. 10, where the apparatus includes:

a first acquiring module 1001, configured to acquire an image of an object to be processed;

a second obtaining module 1002, configured to obtain surface information of an object to be processed based on an image of the object to be processed, where the surface information of the object to be processed includes a reflectivity of the object to be processed and/or a gaussian curvature of the object to be processed;

the processing module 1003 is configured to process surface information of an object to be processed through the model to be trained, so as to obtain characteristics of the object to be processed;

a third obtaining module 1004, configured to obtain a target loss based on the feature of the object to be processed and the real feature of the object to be processed, where the target loss is used to indicate a difference between the feature of the object to be processed and the real feature of the object to be processed;

and an updating module 1005, configured to update parameters of the model to be trained based on the target loss until the model training condition is satisfied, thereby obtaining a feature extraction model.

In a possible implementation manner, the second obtaining module 1002 is configured to calculate an image of the object to be processed based on a photometric stereo algorithm, so as to obtain surface information of the object to be processed.

It should be noted that, because the content of information interaction and execution process between the modules/units of the above-mentioned apparatus is based on the same concept as the method embodiment of the present application, the technical effects brought by the content are the same as the method embodiment of the present application, and specific content may refer to the description in the foregoing illustrated method embodiment of the present application, and will not be repeated herein.

The embodiment of the application also relates to an execution device, and fig. 11 is a schematic structural diagram of the execution device provided by the embodiment of the application. As shown in fig. 11, when the execution device 1100 is a local execution device, it may be specifically represented as an industrial camera, a mobile phone, a tablet, a notebook, a smart wearable device, etc., and when the execution device 1100 is a remote execution device, it may be specifically represented as a cloud server, a web server, an application server, etc., which are not limited herein. The execution device 1100 may be configured with the object surface evaluation device described in the corresponding embodiment of fig. 8 or the statistic acquisition device described in the corresponding embodiment of fig. 9, so as to implement the function of object surface evaluation in the corresponding embodiment of fig. 4. Specifically, the execution apparatus 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (where the number of processors 1103 in the execution device 1100 may be one or more, one processor is exemplified in fig. 11), wherein the processor 1103 may comprise an application processor 11031 and a communication processor 11032. In some embodiments of the application, the receiver 1101, transmitter 1102, processor 1103 and memory 1104 may be connected by a bus or other means.

The memory 1104 may include read-only memory and random access memory and provides instructions and data to the processor 1103. A portion of the memory 1104 may also include non-volatile random access memory (non-volatile random access memory, NVRAM). The memory 1104 stores a processor and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations.

The processor 1103 controls the operation of the execution device. In a specific application, the individual components of the execution device are coupled together by a bus system, which may include, in addition to a data bus, a power bus, a control bus, a status signal bus, etc. For clarity of illustration, however, the various buses are referred to in the figures as bus systems.

The method disclosed in the above embodiment of the present application may be applied to the processor 1103 or implemented by the processor 1103. The processor 1103 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the method described above may be performed by integrated logic circuitry in hardware or instructions in software in the processor 1103. The processor 1103 may be a general purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The processor 1103 can implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 1104, and the processor 1103 reads information in the memory 1104, and in combination with the hardware, performs the steps of the method described above.

The receiver 1101 is operable to receive input numeric or character information and to generate signal inputs related to performing relevant settings and function control of the device. The transmitter 1102 may be used to output numeric or character information through a first interface; the transmitter 1102 may also be configured to send instructions to the disk stack via the first interface to modify data in the disk stack; the transmitter 1102 may also include a display device such as a display screen.

In the embodiment of the present application, when the execution device 1100 is a local execution device, the execution device 1100 further includes a camera, where the camera is electrically connected to the processor 1103, the camera may be provided with a plurality of light sources, and each light source is a multi-wavelength light source, and the camera may implement the shooting operation in the corresponding embodiment of fig. 4 under the control of the processor 1103.

In an embodiment of the present application, in an instance, the processor 1103 is configured to process the surface information of the object through the feature extraction model in the corresponding embodiment of fig. 4, so as to obtain the feature of the object.

The embodiment of the application also relates to training equipment, and fig. 12 is a schematic structural diagram of the training equipment provided by the embodiment of the application. As shown in fig. 12, exercise device 1200 is implemented by one or more servers, exercise device 1200 may vary considerably in configuration or performance, and may include one or more central processing units (central processing units, CPU) 1214 (e.g., one or more processors) and memory 1232, one or more storage media 1230 (e.g., one or more mass storage devices) storing application programs 1242 or data 1244. Wherein memory 1232 and storage medium 1230 can be transitory or persistent. The program stored on storage medium 1230 may include one or more modules (not shown), each of which may include a series of instruction operations for use in training devices. Still further, central processor 1214 may be configured to communicate with storage medium 1230, executing a series of instruction operations in storage medium 1230 on exercise device 1200.

Training apparatus 1200 may also include one or more power sources 1226, one or more wired or wireless network interfaces 1250, one or more input/output interfaces 1258; or one or more operating systems 1241, such as Windows ServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, etc.

Specifically, the training apparatus may perform the model training method in the corresponding embodiment of fig. 7.

The embodiment of the application also relates to a computer storage medium in which a program for performing signal processing is stored which, when run on a computer, causes the computer to perform the steps as performed by the aforementioned performing device or causes the computer to perform the steps as performed by the aforementioned training device.

Embodiments of the present application also relate to a computer program product storing instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned performing device or cause the computer to perform steps as performed by the aforementioned training device.

The execution device, training device or terminal device provided in the embodiment of the present application may be a chip, where the chip includes: a processing unit, which may be, for example, a processor, and a communication unit, which may be, for example, an input/output interface, pins or circuitry, etc. The processing unit may execute the computer-executable instructions stored in the storage unit to cause the chip in the execution device to perform the data processing method described in the above embodiment, or to cause the chip in the training device to perform the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit in the wireless access device side located outside the chip, such as a read-only memory (ROM) or other type of static storage device that may store static information and instructions, a random access memory (random access memory, RAM), etc.

Specifically, referring to fig. 13, fig. 13 is a schematic structural diagram of a chip provided in an embodiment of the present application, where the chip may be represented as a neural network processor NPU 1300, and the NPU 1300 is mounted as a coprocessor on a main CPU (Host CPU), and the Host CPU distributes tasks. The core part of the NPU is an arithmetic circuit 1303, and the controller 1304 controls the arithmetic circuit 1303 to extract matrix data in the memory and perform multiplication.

In some implementations, the arithmetic circuit 1303 includes a plurality of processing units (PEs) inside. In some implementations, the operation circuit 1303 is a two-dimensional systolic array. The arithmetic circuit 1303 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1303 is a general-purpose matrix processor.

For example, assume that there is an input matrix a, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 1302 and buffers the data on each PE in the arithmetic circuit. The arithmetic circuit takes matrix a data from the input memory 1301 and performs matrix operation with matrix B, and the partial result or the final result of the matrix obtained is stored in an accumulator (accumulator) 1308.

Unified memory 1306 is used to store input data and output data. The weight data is directly transferred to the weight memory 1302 through the memory cell access controller (Direct Memory Access Controller, DMAC) 1305. The input data is also carried into the unified memory 1306 through the DMAC.

BIU Bus Interface Unit, bus interface unit 1313, is used for the AXI bus to interact with the DMAC and finger memory (Instruction Fetch Buffer, IFB) 1309.

The bus interface unit 1313 (Bus Interface Unit, abbreviated as BIU) is configured to obtain an instruction from the external memory by the instruction fetch memory 1309, and is also configured to obtain raw data of the input matrix a or the weight matrix B from the external memory by the memory unit access controller 1305.

The DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1306 or to transfer weight data to the weight memory 1302 or to transfer input data to the input memory 1301.

The vector calculation unit 1307 includes a plurality of operation processing units that perform further processing, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and the like, on the output of the operation circuit 1303, if necessary. The method is mainly used for non-convolution/full-connection layer network calculation in the neural network, such as Batch Normalization (batch normalization), pixel-level summation, up-sampling of a predicted label plane and the like.

In some implementations, the vector computation unit 1307 can store the vector of processed outputs to the unified memory 1306. For example, the vector calculation unit 1307 may perform a linear function; alternatively, a nonlinear function is applied to the output of the arithmetic circuit 1303, for example, to linearly interpolate the predicted label plane extracted by the convolution layer, and then, for example, to accumulate a vector of values to generate an activation value. In some implementations, vector computation unit 1307 generates a normalized value, a pixel-level summed value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the arithmetic circuit 1303, for example for use in subsequent layers in a neural network.

An instruction fetch memory (instruction fetch buffer) 1309 connected to the controller 1304 for storing instructions used by the controller 1304;

the unified memory 1306, the input memory 1301, the weight memory 1302, and the finger memory 1309 are all On-Chip memories. The external memory is proprietary to the NPU hardware architecture.

The processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above-mentioned programs.

It should be further noted that the above-described apparatus embodiments are merely illustrative, and that the units described as separate units may or may not be physically separate, and that units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the embodiment of the device provided by the application, the connection relation between the modules represents that the modules have communication connection, and can be specifically implemented as one or more communication buses or signal lines.

From the above description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented by means of software plus necessary general purpose hardware, or of course by means of special purpose hardware including application specific integrated circuits, special purpose CPUs, special purpose memories, special purpose components, etc. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions can be varied, such as analog circuits, digital circuits, or dedicated circuits. However, a software program implementation is a preferred embodiment for many more of the cases of the present application. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk of a computer, etc., comprising several instructions for causing a computer device (which may be a personal computer, a training device, a network device, etc.) to perform the method according to the embodiments of the present application.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Claims

1. A terminal device, characterized in that the terminal device includes: a processor and a camera, the processor is connected to the camera, the camera includes multiple light sources located at different positions, among the multiple light sources Each light source can emit light of multiple wavelengths;

The processor is used to control the camera to activate the plurality of light sources in a preset order to photograph the target object and obtain an image of the target object.

2. The terminal device according to claim 1, wherein the plurality of light sources are arranged in a light source array, and the angle between two adjacent light sources among the plurality of light sources is a preset angle. .

3. The terminal device according to claim 1 or 2, characterized in that the camera meets at least one of the following: the exposure time of the camera is less than or equal to a preset first threshold, and the brightness of the plurality of light sources is greater than or equal to the preset second threshold, the signal-to-noise ratio of the image captured by the camera is greater than or equal to the preset third threshold, and the clarity of the image captured by the camera is greater than or equal to the preset fourth threshold , wherein the signal-to-noise ratio is determined based on wavelengths of light emitted by the plurality of light sources.

4. The terminal device according to any one of claims 1 to 3, characterized in that the processor is also used to:

Obtain surface information of the target object based on the image of the target object, where the surface information of the target object includes the reflectivity of the target object and/or the Gaussian curvature of the target object;

The surface information of the target object is processed through a feature extraction model to obtain the characteristics of the target object;

Based on the characteristics of the target object and the characteristics of the reference object, it is determined that the target object is a surface-qualified object or a surface-unqualified object, and the reference object is a surface-qualified object.

5. The terminal device according to claim 4, wherein the processor is configured to calculate the image of the target object based on a photometric stereo algorithm to obtain the surface information of the target object.

6. The terminal device according to claim 4 or 5, characterized in that the processor is used for:

Obtain the statistics of the characteristics of the reference object, and the statistics of the characteristics of the reference object include the mean vector of the characteristics of the reference object and the covariance matrix of the characteristics of the reference object;

The characteristics of the target object are calculated to obtain the statistics of the characteristics of the target object. The statistics of the characteristics of the target object include the mean vector of the characteristics of the target object and the covariance of the characteristics of the target object. matrix;

Calculate the statistics of the characteristics of the target object and the statistics of the characteristics of the reference object to obtain a score of the target object;

If the score of the target object is greater than the preset fifth threshold, it is determined that the target object is an object with an unqualified surface;

If the score of the target object is less than or equal to the preset fifth threshold, the target object is determined to be a surface-qualified object.

7. The terminal device according to claim 6, characterized in that the processor is used for:

Control the camera to activate the plurality of light sources in the order to photograph the reference object and obtain an image of the reference object;

Obtain surface information of the reference object based on the image of the reference object, where the surface information of the reference object includes the reflectivity of the reference object and/or the Gaussian curvature of the reference object;

The surface information of the reference object is processed through a feature extraction model to obtain the characteristics of the reference object;

The characteristics of the reference object are calculated to obtain the statistics of the characteristics of the reference object. The statistics of the characteristics of the reference object include the mean vector of the characteristics of the reference object and the covariance of the characteristics of the reference object. matrix.

8. The terminal device according to claim 6 or 7, characterized in that the score of the target object is based on the difference between the mean vector of the characteristics of the target object and the mean vector of the characteristics of the reference object, and determining a matrix obtained by fusing the covariance matrix of the characteristics of the target object with the covariance matrix of the characteristics of the reference object.

9. The terminal device according to any one of claims 6 to 8, characterized in that the processor is further configured to: in multiple areas of the target object, assign scores greater than the preset fifth The threshold area is determined as the area in the target object where the surface is unacceptable.

10. A camera method, characterized in that the method is implemented through a terminal device. The terminal device includes: a processor and a camera. The processor is connected to the camera. The camera includes multiple cameras located at different locations. A light source, each of the plurality of light sources can emit light of multiple wavelengths, and the method includes:

The camera is controlled to activate the plurality of light sources in a preset order to photograph the target object and obtain an image of the target object.

11. The method according to claim 10, characterized in that after controlling the camera to activate the plurality of light sources in a preset order to photograph the target object and obtain the image of the target object, the method Also includes:

12. The method according to claim 11, wherein said obtaining the surface information of the target object based on the image of the target object includes:

The image of the target object is calculated based on the photometric stereo algorithm to obtain the surface information of the target object.

13. The method according to claim 11 or 12, characterized in that, based on the characteristics of the target object and the characteristics of the reference object, determining that the target object is a surface-qualified object or a surface-unqualified object includes:

14. The method according to claim 13, characterized in that said obtaining statistics of characteristics of a reference object includes:

15. The method according to claim 13 or 14, characterized in that the score of the target object is based on the difference between the mean vector of the characteristics of the target object and the mean vector of the characteristics of the reference object, and The matrix obtained by fusing the covariance matrix of the characteristics of the target object and the covariance matrix of the characteristics of the reference object is determined.

16. The method according to any one of claims 13 to 15, characterized in that after determining that the target object is an object with an unqualified surface, the method further includes:

Among the multiple areas of the target object, an area with a score greater than the preset fifth threshold is determined to be an area with an unqualified surface in the target object.

17. A computer storage medium, characterized in that the computer storage medium stores a computer program, which when executed by a computer causes the computer to implement the method according to any one of claims 10 to 16.

18. A computer program product, characterized in that the computer program product stores instructions, which when executed by a computer, cause the computer to implement the method according to any one of claims 10 to 16.