WO2024098325A1 - Image processing method, three-dimensional measurement method, and electronic device - Google Patents

Abstract

The present disclosure relates to an image processing method and apparatus, and an electronic device, and relates to the technical field of computers. The image processing method comprises: in response to a first operation by a user in a flowchart, determining the connection relationship between a plurality of algorithm function modules required for executing an image processing task and a plurality of algorithm function modules so as to generate a process of the image processing task, the algorithm function modules being packaged in a plug-in form; parsing a plurality of plug-ins corresponding to the plurality of algorithm function modules to record the connection relationship among the plurality of algorithm function modules; and driving and executing the process of the image processing task according to the recorded connection relationship.

Description

Image processing method, three-dimensional measurement method and electronic device Technical Field

The present disclosure relates to the field of computer technology, and in particular to an image processing method, a three-dimensional measurement method, an information processing method, an electronic device, a non-volatile computer-readable storage medium, and a computer program.

Background technique

With the development of the times, the requirements for product quality are getting higher and higher. The rapid application of new technologies such as AI (Artificial Intelligence) has continuously enhanced product inspection capabilities. At the same time, the application of new technologies such as AI has put all competitors on the same starting line, and industrial visual inspection is ushering in an era of rapid development.

Industrial vision is an important data collection portal for the Industrial Internet and an important part of the Industrial Internet. The industrial production process cannot be separated from the various functional supports of vision. From the storage of production materials, pre-line inspection, quality monitoring and online inspection during production, final product inspection, delivery to overall traceability, all these links cannot be separated from the application of industrial vision.

In the related art, image processing tools are developed independently for specific image processing tasks.

Summary of the invention

According to some embodiments of the present disclosure, there is provided an image processing method, comprising: in response to a first operation of a user in a flowchart, determining multiple algorithm function modules required to perform an image processing task and a connection relationship between the multiple algorithm function modules to generate a process of the image processing task, wherein the algorithm function modules are encapsulated in the form of plug-ins; parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; and driving and executing the process of the image processing task according to the recorded connection relationship.

In some embodiments, the image processing method also includes obtaining at least one newly added algorithm function module in response to the user's second operation in the flowchart; monitoring whether the newly added algorithm module meets the definition specifications; and if it meets the definition specifications, importing the newly added algorithm function module into the process of the image processing task.

In some embodiments, determining multiple algorithm function modules required to perform an image processing task and the connection relationship between the multiple algorithm function modules includes: determining multiple algorithm function modules corresponding to new functions of the host program, and the new functions are used to process image processing tasks; the process of driving and executing image processing tasks includes: using the host program to load multiple plug-ins corresponding to the multiple algorithm function modules, and importing the multiple plug-ins into the host program; calling the host program that has imported multiple plug-ins to drive and execute the process of image processing tasks.

In some embodiments, the plug-in has a common interface defined in the process, and the common interface is used for the plug-in to interact with the process.

In some embodiments, parsing multiple plug-ins corresponding to multiple algorithm function modules includes: after the process is started, monitoring whether there are plug-ins that meet the definition specifications in a preset storage location; if there are plug-ins that meet the definition specifications, parsing the plug-ins.

In some embodiments, parsing the plug-in includes: parsing the plug-in to obtain relevant information of the plug-in; and the process of driving and executing the image processing task includes: loading the plug-in when the relevant information meets the requirements of the process.

In some embodiments, loading the plug-in includes: determining whether the user has permission to call the plug-in; and loading the plug-in if the user has permission to call the plug-in.

In some embodiments, the image processing method further includes: using a process engine function to set input interfaces and output interfaces for multiple plug-ins.

In some embodiments, recording the connection relationship between multiple algorithm function modules includes: recording the connection relationship in XML (eXtensible Markup Language) format to generate relevant information records of the process; according to the recorded connection relationship, driving and executing the process of image processing tasks includes: generating a driver program according to the relevant information records; using the driver program to drive and execute the process of image processing tasks.

In some embodiments, parsing multiple plug-ins corresponding to multiple algorithm function modules includes: parsing description information of the multiple plug-ins, where the description information is used to determine whether the multiple plug-ins meet the definition specifications.

In some embodiments, the plug-in adopts separate storage of the view part and the logic part, and the view part and the logic part have a binding relationship.

In some embodiments, the image processing task includes at least one of a guidance function, a measurement function, a detection function, and a recognition function.

In some embodiments, the guidance function includes at least one of a precise alignment function, an alignment and fitting function, a hand-eye calibration function, or a robot guidance function; the measurement function includes at least one of a distance measurement function, an angle measurement function, a thickness measurement function, or a flatness measurement function; the detection function includes at least one of a defect detection function, a defect classification function, a defect location function, or a flaw detection function; the recognition function includes at least one of a barcode recognition function, a character recognition function, a size recognition function, or a color recognition function.

In some embodiments, each algorithm function module corresponds to each sub-function included in a plurality of image processing algorithms.

In some embodiments, multiple algorithm function modules are selected from candidate algorithm function modules included in multiple image processing function sets, and different image processing function sets correspond to image processing algorithms with different image processing functions.

In some embodiments, the candidate algorithm function module is obtained by decomposing the image processing algorithm corresponding to the image processing function set where the candidate algorithm function module is located into multiple sub-functions.

In some embodiments, the candidate algorithm function module is a DLL (Dynamic Link Library), and the multiple sub-functions are multiple function functions decomposed from the corresponding image processing algorithm, and the DLL is obtained by encapsulating the function functions.

In some embodiments, each image processing function set includes multiple groups, there is an interactive relationship between the candidate algorithm function modules in the same group, there is no interactive relationship between the candidate algorithm function modules in different groups, each group includes multiple layers, and the multiple layers are divided according to the execution order of the image processing algorithms corresponding to the image processing function set to which each group belongs.

In some embodiments, the image processing method further includes: in response to the first operation, determining the hardware access module required for the image processing task and the connection relationship between the hardware access module and multiple algorithm function modules, different hardware access modules are used to access different hardware devices; parsing the plug-in corresponding to the hardware access module to record the connection relationship between the hardware access module and multiple algorithm function modules.

In some embodiments, based on the recorded connection relationship, the process of driving and executing the image processing task includes: executing multiple plug-ins based on the execution relationship type between multiple algorithm function modules in the image processing program, and the execution relationship type includes at least one of serial, parallel, branch execution, conditional execution, and logical judgment.

According to some other embodiments of the present disclosure, a three-dimensional measurement method is provided, including: in response to a first operation of a user in a flowchart, determining multiple algorithm function modules required to perform a three-dimensional measurement task and a connection relationship between the multiple algorithm function modules to generate a process of the three-dimensional measurement task, the algorithm function modules are packaged in the form of plug-ins, and the multiple algorithm function modules include at least one of an image filtering function module, a cross-section function module, or a flatness function module; parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; and driving and executing the process of the three-dimensional measurement task according to the recorded connection relationship.

In some embodiments, the three-dimensional measurement method further includes obtaining at least one newly added algorithm function module in response to the user's second operation in the flowchart; monitoring whether the newly added algorithm module meets the definition specifications; and importing the newly added algorithm function module into the process of the three-dimensional measurement task if it meets the definition specifications.

In some embodiments, determining multiple algorithm function modules required to perform a three-dimensional measurement task and the connection relationship between the multiple algorithm function modules includes: determining multiple algorithm function modules corresponding to new functions of a host program, where the new functions are used to process the three-dimensional measurement task; the process of driving and executing the three-dimensional measurement task includes: using the host program to load multiple plug-ins corresponding to the multiple algorithm function modules, and importing the multiple plug-ins into the host program; calling the host program into which the multiple plug-ins have been imported, and driving and executing the process of the three-dimensional measurement task.

In some embodiments, the three-dimensional measurement method also includes: in response to the first operation, determining the hardware access module required for the three-dimensional measurement task and the connection relationship between the hardware access module and multiple algorithm function modules, different hardware access modules are used to access different hardware devices; parsing the plug-in corresponding to the hardware access module to record the connection relationship between the hardware access module and multiple algorithm function modules.

According to some other embodiments of the present disclosure, there is provided an information processing method, comprising: in response to a first operation of a user in a flowchart, determining multiple algorithm function modules required to perform an information processing task and a connection relationship between the multiple algorithm function modules to generate a process of the information processing task, wherein the algorithm function modules are encapsulated in the form of plug-ins; parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; and driving and executing the process of the information processing task according to the recorded connection relationship.

According to some further embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the image processing method, three-dimensional measurement method or information processing method in any one of the above-mentioned embodiments based on instructions stored in the memory device.

According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, an image processing method, a three-dimensional measurement method, or an information processing method is implemented.

According to some further embodiments of the present disclosure, there is further provided a computer program, comprising: instructions, which, when executed by a processor, cause the processor to execute an image processing method, a three-dimensional measurement method, or an information processing method.

Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

FIG1 is a flowchart showing an image processing method according to some embodiments of the present disclosure;

2a-2b show schematic diagrams of a plug-in framework according to some embodiments of the present disclosure;

FIG3a shows a flowchart of an image processing method according to some other embodiments of the present disclosure;

FIG3 b shows a flow chart of an image processing method according to yet other embodiments of the present disclosure;

FIG3c is a schematic diagram showing an image processing method according to some embodiments of the present disclosure;

FIG4 shows a schematic diagram of an image processing device or an information processing device according to some embodiments of the present disclosure;

FIG5 illustrates a block diagram of some embodiments of an electronic device according to some embodiments of the present disclosure;

FIG6 shows a block diagram of some other embodiments of an electronic device according to some embodiments of the present disclosure;

FIG. 7 shows a schematic diagram of an image processing method according to some other embodiments of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values do not limit the scope of the present disclosure. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, so once a certain item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

The inventor of the present disclosure has found that the above-mentioned related technologies have the following problems: low development efficiency and high cost. In view of this, the present disclosure proposes a technical solution for image processing, which can improve the development efficiency of image processing tools and reduce costs.

As mentioned earlier, the software platform used for processing tool development has the following technical problems at the architectural and functional levels: the scalability is not flexible enough, and the development process is not convenient and efficient when end customers encounter slight adjustments to the process and scenario when using the product; during the algorithm design and programming process, the code needs to be adjusted and complex configurations need to be performed; the deep learning function is not rich enough; when using the deep learning function, image annotation relies on manual work without sample enhancement; and model training relies on powerful local hardware.

Image processing algorithms will be enriched with the continuous development of technology; in addition, combined with numerous application scenarios, visual tools for different scenarios are needed to quickly solve related problems. When the image algorithm changes and different scene requirements are met, the main program needs to be constantly updated, which brings challenges to the flexibility and expansion of the program.

With the application of industrial vision in intelligent production, the scenarios are constantly changing, and customers who use the products also have the need to adjust and improve the products according to different needs. If the product does not adopt a flexible architecture, users will not be flexible enough to use it, which greatly reduces product satisfaction.

Based on the above technical problems, the image processing tool development system disclosed in the present invention fully considers technical convenience and actual customer needs. The characteristics of the development system include lightweight, easy to use, rich functions, strong scalability, process editing, easy to use and operate, and easy to deploy.

In some embodiments, a deep learning software system based on a plug-in architecture and process engine drive is proposed, involving core algorithms and technologies including image processing and deep learning. It mainly solves the various functional supports of industrial vision in the industrial production process. From the warehousing of production materials, pre-line inspection, quality monitoring and online inspection during production, final product inspection, delivery to overall traceability, the application of industrial vision is indispensable, contributing to the intelligent upgrading and development of the manufacturing industry.

For example, the technical solution of the present disclosure can be implemented through the following embodiments.

FIG. 1 shows a flowchart of an image processing method according to some embodiments of the present disclosure.

As shown in FIG1 , in step 110, in response to the user's first operation in the flowchart (such as dragging, pulling, dropping, clicking, etc.), multiple algorithm function modules required to perform the image processing task and the connection relationship between the multiple algorithm function modules are determined to generate the process of the image processing task. For example, the algorithm function module is encapsulated in the form of a plug-in. The core functions that can be implemented by the plug-in may include various image processing algorithms, business implementation logic, etc.

For example, a flowchart can be a template for program development. Based on this template, users can select plug-ins by dragging, dropping, clicking, etc., to form a process containing multiple nodes (encapsulated by plug-ins) in the process editing area of the program development interface; different processes can be generated according to the business to be performed (such as image processing tasks); the host program can record relevant information of the process (such as the number of nodes, execution order, dependencies, etc.).

For example, the lines connecting the nodes in a flowchart are used to represent the relationships between the nodes, including execution order, logical relationships (such as parallel, branching, conditions, etc.), etc.

In some embodiments, a visual scene may require several or even dozens of processes to work together to complete the entire scene. Based on this, a process engine is needed to manage many functions and processes. A process is an instance of a flowchart at runtime. A flowchart can be understood as a "class", and a process can be understood as an "object" driven by a process engine. In this way, it is possible to support the import and export of processes so that they can be referenced in other programs.

In some embodiments, multiple plug-ins are sequentially imported into the same process, and the image processing task is processed using the process. For example, at the plug-in layer, multiple plug-ins are compiled into DLLs, and the DLLs are imported into the same host program using the parsing information of the DLLs.

In some embodiments, the image processing task may include at least one of a guidance function, a measurement function, a detection function, and a recognition function.

For example, the guiding function includes at least one of the precise alignment function, the alignment and fitting function, the hand-eye calibration function or the robot guiding function; the measuring function includes at least one of the distance measurement function, the angle measurement function, the thickness measurement function or the flatness measurement function; the detection function includes at least one of the defect detection function, the defect classification function, the defect location function or the flaw detection function; the recognition function includes at least one of the barcode recognition function, the character recognition function, the size recognition function or the color recognition function.

For example, the interface of the plug-in needs to comply with the interface specifications of the host program and be able to implement the interface functions defined by the host program.

In some embodiments, in response to the user's first operation in the flowchart, multiple algorithm function modules required to perform an information processing task and the connection relationship between the multiple algorithm function modules are determined to generate a process for the information processing task, and the algorithm function modules are encapsulated in the form of plug-ins.

For example, information processing tasks include image processing tasks, guidance functions, measurement functions, detection functions, and recognition functions, etc., and the information processing algorithms corresponding to the information processing tasks include image processing algorithms, deep learning algorithms, etc.

In some embodiments, the information processing algorithm includes a machine learning algorithm. The machine learning model corresponding to the machine learning algorithm is trained by: performing image transformation processing on the annotated first sample image to generate a second sample image; and using the first sample image and the second sample image to train the machine learning model corresponding to the machine learning algorithm.

In some embodiments, the machine learning model corresponding to the machine learning algorithm is trained in the following manner: obtaining a product image containing product defects; generating a background image of the product image; and generating a sample image for training the machine learning model based on the product image and the background image.

In some embodiments, the machine learning model corresponding to the machine learning algorithm is trained in the following manner: the machine learning model corresponding to the machine learning algorithm is trained in the cloud; and the trained machine learning model is sent to the edge to process the image processing task. In some embodiments, in response to the first operation, the hardware access module required for the image processing task and the connection relationship between the hardware access module and the plurality of algorithm function modules are determined, and different hardware access modules are used to access different hardware devices.

In some embodiments, in response to the second operation (such as adding operation) of the user in the flowchart, at least one newly added algorithm function module is obtained. For example, multiple algorithm function modules corresponding to the newly added functions of the host program are determined, and the newly added functions are used to process image processing tasks.

In this way, based on the completed development process (such as a host program with a two-dimensional measurement function), a process with new functions (such as a process with a three-dimensional measurement function) can be formed by adding plug-ins, thereby reducing development costs and improving development efficiency.

In some embodiments, the host program has the ability to display packaged plug-ins and their functions in an interface. After the plug-ins are packaged, they can form nodes in the process; the nodes can be loaded into a list of the process engine (for example, the plug-ins can appear in the list in the form of icons or lists) for the user to select (such as by dragging, pulling, dropping, clicking, etc.).

For example, the host program has the function of encapsulating plug-ins into nodes. The host program has the function of a process engine, which can encapsulate the loaded plug-ins into nodes in each process.

For example, the host program can encapsulate the entire plug-in into a node with an input interface and an output interface in the interface. The output type of the node can be the type of the plug-in (such as text type, script type, program type, etc.), and the input type can be defined in the business function (such as image type or numeric type).

For example, the host program may also have the function of selecting the plug-in by dragging, pulling, dropping, clicking, etc. After selecting the plug-in in the list area, the user may drag it to the process editing area to form a node in the process (for example, having at least one input interface and one output interface).

In some embodiments, the plug-in has a common interface defined in the process, and the common interface is used for the plug-in to interact with the process. For example, the process engine function of the host program is used to set input interfaces and output interfaces for multiple plug-ins.

For example, the host program has the ability to define the interface of the plug-in. The host program can define the common interface of the plug-in to be connected to the host program, which is equivalent to defining a "contract" with the plug-in.

In this way, the plug-in can interact with the host program through the interface defined in the host program to complete the image processing task.

In some embodiments, the overall framework of the present disclosure is a plug-in framework including an algorithm layer at the bottom. The algorithm layer is used to implement multiple information processing algorithms, including core algorithms such as image processing, computer vision, and deep learning. For example, each information processing algorithm is implemented in C++ as a programming language.

In some embodiments, each algorithm function module corresponds to each sub-function included in a plurality of image processing algorithms.

In some embodiments, the plurality of algorithm function modules are selected from candidate algorithm function modules included in a plurality of image processing function sets, and different image processing function sets correspond to image processing algorithms with different image processing functions.

For example, according to the information processing functions that can be realized, multiple information processing algorithms are divided into multiple image processing function sets; in each image processing function set, multiple candidate algorithm function modules are decomposed respectively. For example, the information processing function includes the image processing function.

In some embodiments, the candidate algorithm function module is obtained by decomposing the image processing algorithm corresponding to the image processing function set where the candidate algorithm function module is located into multiple sub-functions.

For example, each image processing function set is divided into multiple sub-functions; and multiple candidate algorithm function modules are decomposed according to the multiple sub-functions. For example, at the algorithm level, the information processing algorithm corresponding to each image processing function set is decomposed into multiple function functions as sub-functions; and the multiple function functions are encapsulated into multiple DLLs as multiple candidate algorithm function modules.

In some embodiments, the candidate algorithm function module is in the form of a DLL, and the multiple sub-functions are multiple functional functions decomposed from the corresponding image processing algorithm, and the DLL is obtained by encapsulating the functional functions.

For example, the functions contained in the information processing algorithm are encapsulated into DLLs to form the underlying algorithm library. DLL is a library containing code and data that can be used by multiple programs at the same time.

In some embodiments, each image processing function set includes multiple groups, there is an interactive relationship between the candidate algorithm function modules in the same group, there is no interactive relationship between the candidate algorithm function modules in different groups, each group includes multiple layers, and the multiple layers are divided according to the execution order of the image processing algorithms corresponding to the image processing function set to which each group belongs.

For example, multiple candidate algorithm function modules are divided into multiple groups. There is no interaction between candidate algorithm function modules belonging to different groups, and candidate algorithm function modules belonging to the same group are divided into multiple layers according to the execution order of the image processing algorithms corresponding to the same group. For example, at least one plug-in is generated according to the grouping and layering of multiple algorithm function modules.

In some embodiments, the plug-in framework includes a plug-in layer. The plug-in layer is used to form a visual tool for processing information processing tasks. For example, the visual tool is composed of a DLL in an underlying algorithm library and associated controls.

In this way, during the product design phase, the product adopts a plug-in structure to ensure the scalability and flexibility of the product.

In some embodiments, the plug-in structure mainly defines the plug-in specifications, plug-in manager, etc. in the framework. For example, the plug-in specifications mainly define the plug-in UI (User Interface) specifications, interface specifications, etc.; the plug-in manager is mainly used for the main functions of plug-in discovery, registration, calling, uninstallation, hot update, etc.

For example, information processing tasks include 3D measurement tasks, and information processing algorithms include 3D measurement algorithms. According to the plug-in specification, create a plug-in class for the 3D measurement algorithm. The plug-in class includes the operating environment definition, plug-in function description, plug-in interface type, etc. defined in the implementation specification.

Implement the basic functions in the plug-in interface, including: discovery, registration, execution, exit, hot start, hot update and other interfaces; develop the interface of the 3D measurement algorithm, including the location and icon that appear in the host program (i.e., main program); develop multiple functional functions included in the 3D measurement algorithm, including: point cloud data processing, height measurement, thickness measurement, flatness measurement, etc.; compile the plug-in class and generate a DLL.

In some embodiments, according to the information processing function to be implemented by the information processing task, an image processing function set corresponding to the information processing task is selected from multiple image processing function sets; and multiple algorithm function modules are selected from multiple candidate algorithm function modules of the corresponding image processing function set.

In some embodiments, according to different algorithm functions, the image processing algorithm can be divided into several image processing function sets, each of which includes multiple sub-functions. For each visual application scenario, multiple sub-functions need to be combined into a complete application process in different combinations and process sequences.

For example, the image processing function set of image processing algorithms includes: hardware access, image preprocessing, image segmentation, image measurement, image recognition, image positioning and other tools.

For example, hardware access can include: multiple sub-functions by camera type (area scan, line scan, etc.), brand (basler/AVT, Hikvision, Delsa), interface (1394/GIGE, USB, CAMLINK), etc. Different cameras have different parameter settings and output formats. After selecting a camera, you need to set the corresponding output. The output of the camera restricts the application of a series of algorithms and operators.

For example, image preprocessing may include: filtering (such as median, mean, Gaussian, low-pass, etc.), chroma processing (RGB, YUV, YIQ, LAB, etc.), morphology (dilation, erosion, opening, closing, etc.), and other sub-functions.

For example, image segmentation may include sub-functions such as edge segmentation (sobel, Canny, level division, level set, etc.), region segmentation (fixed threshold, inter-class difference, OSTU, etc.), etc.

For example, image measurement may include sub-functions such as line finding, circle finding, line fitting, circle fitting, distance measurement, and angle measurement.

For example, image recognition can include sub-functions such as QR code and OCR (optical character recognition).

For example, image positioning may include: BLOB (connected area in the image, used for image binarization), shape matching, edge matching, geometric features and other sub-functions.

For example, other tools may include: glue detection, defect detection, classification, counting and other sub-functions.

In some embodiments, when using the above information processing algorithm and image processing function set, it is necessary to check the execution relationship type between modules. For example, the image processing flow includes serial, parallel, branch, conditional execution, logical judgment and other processes. These processes need to be judged and executed by the process engine.

In some embodiments, a new process is first created, that is, this process is currently an empty process that does not contain any nodes, and is named, and the save path and whether to encrypt and permissions are set; in this empty process, a processing flow is formed by dragging and dropping each functional module in the toolbox of the main program.

For example, in the function module, select the image filter function module to filter out some noise points in the laser acquisition process, so that the data formed is smoother and is conducive to subsequent processing. For example, when selecting the filter tool, you need to select the image filter function module that supports the depth map, otherwise after dragging other tools, when connecting the output of the 3D camera function tool to the filter function tool, an error will occur, prompting that the type does not match, etc.

For example, when adding a cross-sectional functional module, obtain the cut-out images of each interface of the product to be processed, calculate the flatness, and verify the matching of the cross-sectional functional module.

For example, before adding a cross-section function module, a logic function module can be added to determine whether the product supports 3D point cloud data and cross-section tools, etc.

For example, the output of the cross-section function module is connected to the input of the flatness function module; the output of the flatness function module is connected to the final result output to form a processing flow.

For example, the process engine controls the operation of the entire process. If there are multiple processes, it is necessary to consider computer resources such as memory, CPU (central processing unit) and process optimization; the entire process can be exported as a ZIV file.

As a process file, ZIV file can be imported into other projects through encryption and serialization. It also supports secondary development and provides access to mainstream programming languages.

For example, the process engine controls the output and execution smoothness, logic, etc. of each functional tool.

In step 120, multiple plug-ins corresponding to multiple algorithm function modules are parsed to record the connection relationship between the multiple algorithm function modules.

In some embodiments, after the process is started, it is monitored whether there is a plug-in that meets the definition specifications in the preset storage location; if there is a plug-in that meets the definition specifications, the plug-in is parsed.

For example, the host program has a plug-in discovery function. After the host program is started, the host program has the function of independently monitoring and discovering whether there is a plug-in that meets the host program definition specification in a specified location (for example, a folder \\PlugIn that can be specified for the host program); if a new plug-in that meets the host program definition specification is found, the plug-in is parsed.

In some embodiments, the plug-in is parsed to obtain relevant information of the plug-in so that the plug-in can be loaded if the relevant information meets the requirements of the process (such as step 130).

For example, the plug-in related information may include description information of the plug-in, such as the plug-in name, type, version, creator, etc. When the host program performs parsing, the description information is first parsed; then, based on the description information, it is determined whether the plug-in complies with the host program specification.

In some embodiments, the connection relationship is recorded in XML format to generate relevant information records of the process; a driver is generated based on the relevant information records; and the driver is used to drive and execute the process of the image processing task. For example, the description information of multiple plug-ins is parsed, and the description information is used to determine whether the multiple plug-ins meet the definition specifications.

For example, the generated DLL and its description files, such as XML files, are copied to the designated file directory (such as the generally defined Plugin) of the host (ie, the original main program).

For example, when the hot update function is implemented, after the plug-in file is placed in the specified file directory, the host program actively discovers the new plug-in file; after discovering the plug-in file, the host program parses the description information in the XML file; if it meets the interface requirements defined by this product, the host program initiates loading of the corresponding DLL file, and then determines whether the DLL file is qualified; starts the loading function and performs the registration process; imports the interface, algorithm and other content into the main program according to the content implemented in the specification.

For example, the main program system will automatically update and display the new 3D measurement module in the function list of the main program according to the position or icon defined in the standard interface during design; the new 3D measurement module can realize the relevant functions of the new 3D measurement scene according to the customer's requirements.

For example, the host program has the ability to create a process engine and record the relationships between plug-ins in the process engine (such as execution order, logical relationships, type matching judgment, etc.). In the host program, it supports the creation of a process (for example, the process includes input images, business processing of images, final output, and supported logical and geometric operations, etc.). When dragging plug-ins, the host program needs to record the number of dragged plug-ins, execution order, dependency relationships, type judgment, etc.

In some embodiments, the main program is used to determine whether the interface of the DLL meets the interface requirements based on the parsed information; if it meets the interface requirements, the main program is used to import the DLL into the main program through a registration process.

For example, the host program has the function of parsing plug-ins. When a plug-in file (such as a DLL file) is found, the host program needs to parse the plug-in and obtain relevant information about the plug-in. For example, the relevant information may include the basic information of the parsed plug-in (such as name, version, type, creator, whether the interface implementation is complete, etc.). If the relevant information meets the requirements of the host program, the plug-in is loaded.

In some embodiments, the plug-in corresponding to the hardware access module is parsed to record the connection relationship between the hardware access module and the plurality of algorithm function modules.

In some embodiments, at the plug-in layer, at least one plug-in is generated based on multiple algorithm function modules and their associated controls. For example, the controls are stored in a manner of separating data from logic, and the controls have interfaces.

In some embodiments, the plug-in adopts separate storage of the view part and the logic part, and the view part and the logic part have a binding relationship. For example, the plug-in can be stored visually, including the interface, input parameters, output results, icon form, location, etc. The input parameters and output results can also be edited to facilitate code-free parameter setting of the plug-in in the process engine. When the host program loads the plug-in, the form and appearance of the plug-in can be defined.

For example, the algorithm function modules decomposed from the underlying algorithm exist in the form of DLL, and the control uses MVVM (Model-View-ViewMode) programmed by WPF (Windows Presentation Foundation) to separate the View (view) and Model (logic), and interact through ViewModel; the entire plug-in (DLL and its associated controls) is encapsulated in the form of PlugIn.

In some embodiments, a hardware access module corresponding to the information processing task is selected according to the information processing task. Different hardware access modules are used to access different hardware devices; and at least one plug-in is generated according to the hardware access module and multiple algorithm function modules.

For example, when selecting hardware, select a profilometer from a 3D camera among multiple hardware; set the parameters of the profilometer, including field of view, exposure time, and output file format: including grayscale, depth map, 3D point cloud data, etc.

In step 130, the process of the image processing task is driven and executed according to the recorded connection relationship. For example, the plug-in adopts a key-value pair structure. The key of the key-value pair structure includes the interface of the plug-in, and the value of the key-value pair structure includes the name of the plug-in.

In some embodiments, it is monitored whether the newly added algorithm module meets the definition specification; if it meets the definition specification, the newly added algorithm function module is imported into the process of the image processing task. For example, multiple plug-ins corresponding to multiple algorithm function modules are loaded by the host program, and the multiple plug-ins are imported into the host program; the host program with the imported multiple plug-ins is called to drive and execute the process of the image processing task.

In some embodiments, it is determined whether the user has the authority to call the plug-in; if the user has the authority to call the plug-in, the plug-in is loaded.

For example, the host program has the function of loading plug-ins. The host program obtains relevant information of the plug-in by parsing the plug-in; the host program checks whether it has the authority to load the plug-in based on the relevant information (for example, the host program can limit the number, type, function, etc. of plug-ins that can be loaded through authority management); if it has the authority, the plug-in is loaded to load all functions of the plug-in (such as interface functions, business functions, visualization functions, etc.).

For example, the host program has the function of managing plug-ins. The host program has the function of managing loaded plug-ins, including adding them to the host program plug-in list, permission management, process call related information, etc.

For example, the host program has the function of uninstalling plug-ins. When the host program exits or in the plug-in management, the loaded plug-in can be uninstalled.

In some embodiments, multiple plug-ins are executed according to the execution relationship type between multiple algorithm function modules in the image processing program, and the execution relationship type includes at least one of serial, parallel, branch execution, conditional execution, and logical judgment.

For example, the host program has the function of saving the entire process information of the process engine. The host layer can record the process-related information and generate the execution driver of the process engine according to the recorded information.

For example, the host program has the function of importing and exporting process information. The host program supports saving and exporting processes and their information, and also supports importing process information to form new processes.

FIG. 2 a shows a schematic diagram of a plug-in framework according to some embodiments of the present disclosure.

As shown in FIG. 2 a , the overall architecture is a plug-in framework, which may include interactive configuration functions, core functions, and basic functions.

For example, the interactive configuration function may include an interface interaction system, system configuration management, etc.; the core function may include multiple image processing function sets, each image processing function set may include multiple sub-functions; the basic functions may include a core underlying algorithm library with multiple information processing algorithms, data access, communication management, accelerated computing, etc.

The core of the overall architecture includes the underlying algorithm library and four derived functional partitions (such as guidance functional partition, measurement functional partition, detection functional partition and identification functional partition). The overall architecture fully considers the subsequent extensibility of UI and functions (adopting plug-in structure design, which can be flexibly edited according to specifications). When dealing with different application scenarios, flexible tool chain programming (internal logic inspection, process recording, etc.) is used to export projects to adapt to different platforms and development language environment calls.

For example, the basic functions of the overall architecture include core underlying algorithm library, data access, communication management, accelerated computing, etc. For example, the core underlying algorithm library mainly includes core algorithms for image processing, computer vision, deep learning, etc., using C++ as the programming language, and encapsulating the corresponding algorithm functions into dll to form the underlying algorithm library.

For example, the core functions of the overall architecture include assembling various algorithm function modules decomposed from the algorithm in the underlying algorithm library to achieve various functions, that is, visual tools formed by assembling various algorithm function modules. The visual tool is composed of dll and matching controls in the underlying algorithm library. The underlying algorithm exists in the form of dll, and the control uses WPF programming MVVM to separate View and ViewModel, and the entire plug-in is encapsulated in the form of PlugIn.

For example, the interactive configuration function of the overall architecture, that is, the entire main program framework, mainly includes the interface interaction system, system configuration management, etc.

For example, the main program framework can also include a plug-in manager, non-functional requirements, common functions, overall UI design, etc. The plug-in manager is used to discover plug-in objects, load, unload, layout and schedule plug-ins; non-functional requirements include settings, systems, help, etc.; common functions include communication modules, process execution, result display, project import and export, etc.; the overall UI design includes the overall UI of the MVVM architecture design except for plug-ins.

In some embodiments, the framework includes a plug-in structure. In order to avoid complex system interactions, redundant designs, and difficult maintenance, the overall UI uses WPF interface technology and .NET Framework 4.0. The main framework of the plug-in structure contains different class libraries; the UI control uses the MVVM data structure and uses data binding to separate the interface from the logic; all external open source controls used have their interfaces abstracted, so other controls with the same function can be used at any time without changing the upper-level code.

For example, the structure of the plugin adopts the storage method of key-value pairs. The key includes the interface that the plugin is required to implement, and the value includes the corresponding plugin name dictionary. The plugin manager can insert and delete (with security risks) data in the plugin dictionary at any time.

In some embodiments, the framework includes a process engine algorithm assembler. The complex information processing algorithm is decomposed into different algorithm function modules and then reassembled. Therefore, by simply designing different algorithm function modules and defining their interfaces, users can assemble and configure the modules according to different information processing tasks to solve complex technical problems in the development of actual processing tools.

For example, an algorithm function module can be understood as a box with input and output interfaces. An algorithm function module may contain multiple inputs or multiple outputs. Only when all inputs are satisfied can the corresponding algorithm be executed, but it may not be output to other algorithm function modules. Multiple algorithm function modules can be assembled into an algorithm network in a complex form to achieve automated processing.

For example, there may be coupling and stratification between different algorithm function modules. The algorithm management system will implement stratification and grouping of modules. Algorithm function modules between different groups have no interaction, so they can be executed in parallel; algorithm function modules in the same group are executed from the first layer to the lower layer in sequence.

FIG. 2 b shows a schematic diagram of a plug-in framework according to some embodiments of the present disclosure.

As shown in FIG. 2b , the working process driven by the plug-in framework and the process engine includes the following steps.

For example, start the main program (.exe); start the authorization detection thread (always working, regularly detecting whether the authorization exists). The authorization thread reads the authorization permission and calls the corresponding configuration (mainly the plug-in function enablement flag).

For example, the authorization process may include two parts: the user's access rights to the various functions of the overall framework (such as various interactive configuration functions, various basic functions, etc.); and the user's access rights to the algorithm function module.

For example, all functions of the main program are started, including the plug-in manager, non-core functions, communication and general functions; the plug-in manager loads the corresponding plug-in according to the authorized configuration information. For example, if there is an abnormality, an abnormality alarm is issued. After loading is completed, the corresponding visual tools and modules are displayed in the main program interface.

For example, business processing (such as visual engineering programming, etc.); generating process sequences. For example, you can create new functions in the process editing area, perform process programming to form visual engineering; the system records the process in XML in real time to form process records.

For example, when saving the engineering process, process records, visual tool parameter settings, images, files and other contents are packaged together for serialization and saved as a custom format file to form a visual engineering plan.

For example, when closing a software process, the system automatically uninstalls the plug-in, stops authorization inspection, and exits the software.

For example, when loading a project plan, the project is analyzed first. The process record file of the read content mainly includes the number, sequence, dependencies, etc. of the visual tools involved; the parameters set for each visual tool are read and restored; the images and files involved are read and restored; and the entire project is restored in the software.

For example, in the model verification process, the model is verified by MD5 (Message-Digest Algorithm) to ensure that there are no problems in the data transmission process; the model's configuration information is checked to see if it matches the current business to ensure that the algorithm model does match the current business.

FIG. 3 a shows a flowchart of an image processing method according to some other embodiments of the present disclosure.

As shown in FIG. 3 a , in step 310 a , a solution (ie, a processing flow) related to an image processing task is loaded.

In step 320a, the plan is edited to modify the parameters therein.

In step 330a, the edited plan is saved.

In some embodiments, the scheme is maintained through steps 305a to 335a. In step 305a, the corresponding process control is initialized. In step 315a, an empty scheme is created. In step 325a, the scheme is built and edited. In step 335a, the process is disabled or deleted.

In step 340a, the saved scheme is executed once; or in step 345a, the saved scheme is executed continuously, and in step 355, the execution is stopped.

In step 350a, the results of each algorithm function module are obtained.

In step 360a, the protocol is closed.

In some embodiments, take the 3D measurement task as an example. The delivered product is a tool for customers to implement 2D dimension measurement functions. However, as customer products and needs change, customers need to add 3D dimension measurement functions to existing products. That is, it includes not only length, width, distance measurement, etc., but also thickness, flatness, and slope angle measurement. This requires adding relevant algorithms and tool modules to the original product. Customers want to expand related functions without changing the operation of the existing program.

For example, the new functions required for the above-mentioned 3D measurement tasks include a new UI interface, an algorithm adapted to 3D measurement, and the newly added 3D measurement algorithm can be compatible and linked with the original algorithm module.

The new algorithm function module needs to follow the product's plug-in specifications to design and define the interface; the new plug-in needs to use an independent class that can be discovered by the main program (host program) and loaded using the class loader.

For example, according to the plug-in specification, a plug-in class for a 3D measurement algorithm is created. The plug-in class includes the operating environment definition, plug-in function description, plug-in interface type, etc. defined in the implementation specification.

Implement basic functions in the plug-in interface, including: discovery, registration, execution, exit, hot start, hot update and other interfaces; develop the interface of the 3D measurement algorithm, including the location and icon of the host program; develop multiple functional functions included in the 3D measurement algorithm, including: point cloud data processing, height measurement, thickness measurement, flatness measurement, etc.; compile the plug-in class and generate DLL.

For example, the generated DLL and its description files, such as XML files, are copied to the designated file directory (such as the generally defined Plugin) of the host (ie, the original main program).

For example, when the hot update function is implemented, after the plug-in file is placed in the specified file directory, the host program actively discovers the new plug-in file; after discovering the plug-in file, the host program parses the description information in the XML file; if it meets the interface requirements defined by this product, the host program initiates loading of the corresponding DLL file, and then determines whether the DLL file is qualified; starts the loading function and performs the registration process; imports the interface, algorithm and other content into the main program according to the content implemented in the specification.

For example, the main program system will automatically update and display the new 3D measurement module in the function list of the main program according to the position or icon defined in the standard interface during design; the new 3D measurement module can realize the relevant functions of the new 3D measurement scene according to the customer's requirements.

FIG. 3 b shows a flowchart of an image processing method according to yet other embodiments of the present disclosure.

As shown in FIG3b, in step 310b, the solution is loaded. For example, before executing step 310b, step 305b may be executed to initialize the process control. After executing step 310b, step 315b may be executed to edit the solution and modify parameters.

In step 320b, an algorithm function module is obtained.

In step 330b, the parameter object of the algorithm function module is obtained.

In step 340b, the parameters of the algorithm function module are set and obtained.

In some embodiments, after executing 320b, step 325b may be executed to bind the algorithm function module with the corresponding parameter configuration control; and step 328b may be executed to edit the parameter configuration control and modify the parameters.

In step 350b, the solution is saved.

FIG. 3 c shows a schematic diagram of an image processing method according to some embodiments of the present disclosure.

As shown in FIG. 3c , still taking the above 3D measurement task as an example, the image processing method may include the following process.

For example, first create a new process, name it, set the save path, whether to encrypt, and permissions; in this empty process, drag and drop each functional module in the toolbox of the main program to form a processing flow.

For example, when selecting hardware, select the profilometer in the 3D camera as the image source among multiple hardware; set the parameters of the profilometer, including field of view, exposure time, output file format: including grayscale, depth map, 3D point cloud data, etc.

For example, in the function module, select the image filter function module to filter out some noise points in the laser acquisition process, so that the data formed is smoother and is conducive to subsequent processing. For example, when selecting the filter tool, you need to select the image filter function module that supports the depth map, otherwise after dragging other tools, when connecting the output of the 3D camera function tool to the filter function tool, an error will occur, prompting that the type does not match, etc.

For example, when adding a cross-sectional functional module, obtain the cut-out images of each interface of the product to be processed, calculate the flatness, and verify the matching of the cross-sectional functional module.

For example, before adding a cross-section function module, a logic function module can be added to determine whether the product supports 3D point cloud data and cross-section tools, etc.

For example, the output of the cross-section function module is connected to the input of the flatness function module; the output of the flatness function module is connected to the final result output to form a processing flow.

For example, you can also select algorithm function modules for color binarization, grayscale weight processing, and caliper tools to add to the processing flow.

For example, the process engine controls the operation of the entire process. If there are multiple processes, it is necessary to consider the computer resources such as memory, CPU and process optimization; the entire process can be exported as a ZIV file.

As a process file, ZIV file can be imported into other projects through encryption and serialization. It also supports secondary development and provides access to mainstream programming languages.

For example, the process engine controls the output and execution smoothness, logic, etc. of each functional tool.

FIG. 4 shows a schematic diagram of an image processing device or an information processing device according to some embodiments of the present disclosure.

As shown in FIG4 , the image processing device 4 includes: a selection unit 41, which is used to select a plurality of algorithm function modules corresponding to the image processing task according to the image processing task, and the plurality of corresponding algorithm function modules are obtained by decomposing the corresponding image processing algorithm of the image processing task; a generation unit 42, which is used to generate a plurality of plug-ins according to the plurality of corresponding algorithm function modules; and a processing unit 43, which is used to process the image processing task using the plurality of plug-ins.

In some embodiments, the image processing device 4 also includes: a decomposition unit 44, which is used to divide multiple image processing algorithms into multiple image processing function sets according to the image processing functions that can be realized, and decompose multiple candidate algorithm function modules in each image processing function set; a selection unit 41 selects an image processing function set corresponding to the image processing task from multiple image processing function sets according to the image processing function to be realized by the image processing task, and selects multiple algorithm function modules from multiple candidate algorithm function modules in the corresponding image processing function set.

In some embodiments, the decomposition unit 44 divides each image processing function set into a plurality of sub-functions, and decomposes a plurality of candidate algorithm function modules according to the plurality of sub-functions.

In some embodiments, the decomposition unit 44 decomposes the image processing algorithm corresponding to each image processing function set into multiple functional functions as sub-functions at the algorithm layer, and encapsulates the multiple functional functions into multiple DLLs as multiple candidate algorithm function modules.

In some embodiments, multiple candidate algorithm function modules are divided into multiple groups, there is no interactive relationship between the candidate algorithm function modules belonging to different groups, and the candidate algorithm function modules belonging to the same group are divided into multiple layers according to the execution order of the corresponding image processing algorithms of the same group; the generation unit 42 generates at least one plug-in according to the grouping and layering of the multiple algorithm function modules.

In some embodiments, the generation unit 42 generates at least one plug-in at the plug-in layer according to a plurality of algorithm function modules and their associated controls.

In some embodiments, the image processing device 4 further includes: a storage unit 45 for storing controls in a manner of separating data from logic, and the controls have an interface.

In some embodiments, at least one plug-in includes multiple plug-ins, and the processing unit 43 imports the multiple plug-ins into the same processing flow in sequence, and processes the image processing task using the processing flow.

In some embodiments, the processing unit compiles multiple plug-ins into DLLs at the plug-in layer; and uses the parsing information of the DLLs to import the DLLs into the same main program.

In some embodiments, the processing unit 43 uses the main program to determine whether the interface of the DLL meets the interface requirements based on the parsing information. If it meets the interface requirements, the main program is used to import the DLL into the main program through a registration process.

In some embodiments, the plug-in adopts a key-value pair structure, the key of the key-value pair structure includes the interface of the plug-in, and the value of the key-value pair structure includes the name of the plug-in.

In some embodiments, the selection unit 41 selects a hardware access module corresponding to the image processing task according to the image processing task, and different hardware access modules are used to access different hardware devices; the generation unit 42 generates at least one plug-in according to the hardware access module and multiple algorithm function modules.

In some embodiments, the image processing algorithm includes an image processing algorithm based on a machine learning model, and the image processing tasks include guidance function, measurement function, detection function and recognition function.

In some embodiments, the machine learning model is trained in the following manner: performing image transformation processing on the labeled first sample image to generate a second sample image; and using the first sample image and the second sample image to train the machine learning model corresponding to the machine learning algorithm.

In some embodiments, the machine learning model corresponding to the machine learning algorithm is trained in the following manner: obtaining a product image containing product defects; generating a background image of the product image; and generating a sample image for training the machine learning model based on the product image and the background image.

In some embodiments, the machine learning model corresponding to the machine learning algorithm is trained in the following manner: the machine learning model corresponding to the machine learning algorithm is trained in the cloud; and the trained machine learning model is sent to the edge to process the image processing task.

In some embodiments, the information processing device 4 includes: a selection unit 41, used to select multiple algorithm function modules corresponding to the information processing task according to the information processing task, and the multiple algorithm function modules are obtained by decomposing multiple information processing algorithms; a generation unit 42, used to generate at least one plug-in according to the multiple algorithm function modules; a processing unit 43, used to use at least one plug-in to process the information processing task.

In some embodiments, the information processing device 4 also includes: a decomposition unit 44, which is used to divide multiple information processing algorithms into multiple image processing function sets according to the information processing functions that can be realized, and decompose multiple candidate algorithm function modules in each image processing function set; a selection unit 41 selects an image processing function set corresponding to the information processing task from multiple image processing function sets according to the information processing function to be realized by the information processing task, and selects multiple algorithm function modules from multiple candidate algorithm function modules of the corresponding image processing function set.

In some embodiments, the decomposition unit 44 divides each image processing function set into a plurality of sub-functions, and decomposes a plurality of candidate algorithm function modules according to the plurality of sub-functions.

In some embodiments, the decomposition unit 44 decomposes the information processing algorithm corresponding to each image processing function set into multiple functional functions as sub-functions at the algorithm layer, and encapsulates the multiple functional functions into multiple DLLs as multiple candidate algorithm function modules.

In some embodiments, multiple candidate algorithm function modules are divided into multiple groups, there is no interactive relationship between the candidate algorithm function modules belonging to different groups, and the candidate algorithm function modules belonging to the same group are divided into multiple layers according to the execution order of the corresponding information processing algorithms of the same group; the generation unit 42 generates at least one plug-in according to the grouping and layering of the multiple algorithm function modules.

In some embodiments, the generation unit 42 generates at least one plug-in at the plug-in layer according to a plurality of algorithm function modules and their associated controls.

In some embodiments, the information processing device 4 further includes: a storage unit 45 for storing controls in a manner of separating data from logic, and the controls have an interface.

In some embodiments, at least one plug-in includes multiple plug-ins, and the processing unit 43 imports the multiple plug-ins into the same processing flow in sequence, and processes the information processing task using the processing flow.

In some embodiments, the processing unit compiles multiple plug-ins into DLLs at the plug-in layer; and uses the parsing information of the DLLs to import the DLLs into the same main program.

In some embodiments, the processing unit 43 uses the main program to determine whether the interface of the DLL meets the interface requirements based on the parsing information. If it meets the interface requirements, the main program is used to import the DLL into the main program through a registration process.

In some embodiments, the plug-in adopts a key-value pair structure, the key of the key-value pair structure includes the interface of the plug-in, and the value of the key-value pair structure includes the name of the plug-in.

In some embodiments, the selection unit 41 selects a hardware access module corresponding to the information processing task according to the information processing task, and different hardware access modules are used to access different hardware devices; the generation unit 42 generates at least one plug-in according to the hardware access module and multiple algorithm function modules.

In some embodiments, the information processing algorithm includes an image processing algorithm based on a machine learning model, and the information processing tasks include guidance function, measurement function, detection function and recognition function.

In some embodiments, the machine learning model is trained in the following manner: performing image transformation processing on the labeled first sample image to generate a second sample image; and using the first sample image and the second sample image to train the machine learning model corresponding to the machine learning algorithm.

In some embodiments, the machine learning model is trained by: acquiring a product image containing a product defect; generating a background image of the product image; and generating a sample image for training the machine learning model based on the product image and the background image.

In some embodiments, the machine learning model is trained in the following manner: training the machine learning model corresponding to the machine learning algorithm in the cloud; and sending the trained machine learning model to the edge to process the image processing task.

FIG. 5 shows a block diagram of some embodiments of an electronic device according to some embodiments of the present disclosure.

As shown in FIG5 , the electronic device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51 , and the processor 52 is configured to execute the image processing method or information processing method in any embodiment of the present disclosure based on instructions stored in the memory 51 .

The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

FIG. 6 shows a block diagram of further embodiments of an electronic device according to some embodiments of the present disclosure.

As shown in FIG6 , the electronic device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610 , and the processor 620 is configured to execute the image processing method or information processing method in any one of the aforementioned embodiments based on instructions stored in the memory 610 .

The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

The electronic device 6 may also include an input/output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, the memory 610, and the processor 620 may be connected, for example, via a bus 660. The input/output interface 630 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, a touch screen, a microphone, and a speaker. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

FIG. 7 shows a schematic diagram of an image processing method according to some other embodiments of the present disclosure.

As shown in FIG7 , the development of the processing flow of the business is plug-in-based, which is a dynamically extended software development architecture and a program written using a standardized application program interface. The plug-in is mainly implemented by relying on interfaces (such as business interfaces and plug-in interfaces) and their reflection mechanisms. The interface is actually a "contract". The host program 71 uses this "contract" to constrain whether there is an object that meets the expectations (such as plug-in 72). If it does not meet the requirements, the object will not be loaded.

Plug-in 72 is an object shared through dynamic links. It is a modular component with functions that can be reused and flexibly managed (such as maintenance, replacement, addition, deletion, etc.). Based on the extensibility of plug-in 72, the independence and decoupling of different functional modules of the business are realized, which increases the maintainability and extensibility of program development.

The plug-in 72 can add new functions on the basis of the existing host program 71 without destroying its existing core functions. In this way, third-party developers can add value to the existing system configuration, and other developers can also cooperate with each other in collaborative development. Through multiple plug-ins 72, the focus points in the program development process can be processed separately, the implementation details can be hidden, and independent testing of each focus point can be performed. It should be understood by those skilled in the art that the embodiments of the present disclosure can be provided as methods, systems, or computer program products. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims (28)

An image processing method, comprising: In response to a first operation of a user in the flowchart, a plurality of algorithm function modules required to perform an image processing task and a connection relationship between the plurality of algorithm function modules are determined to generate a process of the image processing task, wherein the algorithm function modules are packaged in a plug-in form; Parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; According to the recorded connection relationship, the process of the image processing task is driven and executed. The image processing method according to claim 1, further comprising In response to a second operation of the user in the flowchart, acquiring at least one newly added algorithm function module; Monitoring whether the newly added algorithm module complies with the defined specifications; In the case of meeting the definition specifications, the newly added algorithm function module is introduced into the process of the image processing task. The image processing method according to claim 1, wherein the determining of the multiple algorithm function modules required to perform the image processing task and the connection relationship between the multiple algorithm function modules comprises: Determine a plurality of algorithm function modules corresponding to newly added functions of the host program, wherein the newly added functions are used to process the image processing task; The process of driving and executing the image processing task includes: Using the host program to load multiple plug-ins corresponding to the multiple algorithm function modules, and importing the multiple plug-ins into the host program; The host program into which the plurality of plug-ins are imported is called to drive and execute the process of the image processing task. The image processing method according to claim 1, wherein the plug-in has a common interface defined in the process, and the common interface is used for the plug-in to interact with the process. The image processing method according to claim 1, wherein the parsing of the plurality of plug-ins corresponding to the plurality of algorithm function modules comprises: After the process is started, monitoring whether there is a plug-in that meets the defined specifications in a preset storage location; If there is a plug-in that meets the definition specification, the plug-in is parsed. The image processing method according to claim 5, wherein parsing the plug-in comprises: Parsing the plug-in to obtain relevant information of the plug-in; The process of driving and executing the image processing task includes: When the relevant information meets the requirements of the process, the plug-in is loaded. The image processing method according to claim 6, wherein the loading the plug-in comprises: Determine whether the user has permission to call the plug-in; If the user has the permission to call the plug-in, the plug-in is loaded. The image processing method according to claim 1, further comprising: The process engine function is used to set input interfaces and output interfaces for the multiple plug-ins. The image processing method according to claim 1, wherein the recording of the connection relationship between the plurality of algorithm function modules comprises: Recording the connection relationship in an extensible markup language (XML) format to generate relevant information records of the process; The process of driving and executing the image processing task according to the recorded connection relationship includes: Generate a driver program according to the relevant information records; The driver is used to drive and execute the process of the image processing task. The image processing method according to claim 1, wherein the parsing of the plurality of plug-ins corresponding to the plurality of algorithm function modules comprises: The description information of the plurality of plug-ins is parsed, where the description information is used to determine whether the plurality of plug-ins conform to a definition specification. The image processing method according to claim 1, wherein the plug-in adopts separate storage of a view part and a logic part, and the view part and the logic part have a binding relationship. The image processing method according to claim 1, wherein the image processing task includes at least one of a guiding function, a measuring function, a detecting function and a recognizing function. The image processing method according to claim 12, wherein: The guiding function includes at least one of a precise positioning function, a positioning and fitting function, a hand-eye calibration function or a robot guiding function; The measurement function includes at least one of a distance measurement function, an angle measurement function, a thickness measurement function or a flatness measurement function; The detection function includes at least one of a defect detection function, a defect classification function, a defect location function or a flaw detection function; The recognition function includes at least one of a barcode recognition function, a character recognition function, a size recognition function or a color recognition function. According to the image processing method according to any one of claims 1 to 11, each algorithm function module corresponds to each sub-function included in the multiple image processing algorithms. The image processing method according to claim 14, wherein: The multiple algorithm function modules are selected from candidate algorithm function modules included in multiple image processing function sets, and different image processing function sets correspond to image processing algorithms with different image processing functions. According to the image processing method according to claim 15, wherein the candidate algorithm function module is obtained by decomposing the image processing algorithm corresponding to the image processing function set where the candidate algorithm function module is located into multiple sub-functions. According to the image processing method according to claim 16, wherein the candidate algorithm function module is a dynamic link library DLL, the multiple sub-functions are multiple function functions decomposed from the corresponding image processing algorithm, and the DLL is obtained by encapsulating the function function. According to the image processing method of claim 15, each image processing function set includes multiple groups, there is an interactive relationship between the candidate algorithm function modules in the same group, there is no interactive relationship between the candidate algorithm function modules in different groups, each group includes multiple layers, and the multiple layers are divided according to the execution order of the image processing algorithm corresponding to the image processing function set to which the group belongs. The image processing method according to any one of claims 1 to 11, further comprising: In response to the first operation, determining a hardware access module required for the image processing task and a connection relationship between the hardware access module and the plurality of algorithm function modules, wherein different hardware access modules are used to access different hardware devices; The plug-in corresponding to the hardware access module is parsed to record the connection relationship between the hardware access module and the multiple algorithm function modules. The image processing method according to any one of claims 1 to 11, wherein the process of driving and executing the image processing task according to the recorded connection relationship comprises: The multiple plug-ins are executed according to the execution relationship type between the multiple algorithm function modules in the image processing program, and the execution relationship type includes at least one of serial, parallel, branch execution, conditional execution, and logical judgment. A three-dimensional measurement method, comprising: In response to a first operation of the user in the flowchart, a plurality of algorithm function modules required for performing a three-dimensional measurement task and a connection relationship between the plurality of algorithm function modules are determined to generate a process of the three-dimensional measurement task, wherein the algorithm function modules are packaged in a plug-in form, and the plurality of algorithm function modules include at least one of an image filtering function module, a cross-section function module, or a flatness function module; Parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; According to the recorded connection relationship, the process of the three-dimensional measurement task is driven and executed. The three-dimensional measurement method according to claim 21, further comprising In response to a second operation of the user in the flowchart, acquiring at least one newly added algorithm function module; Monitoring whether the newly added algorithm module complies with the defined specifications; In the case of meeting the definition specifications, the newly added algorithm function module is introduced into the process of the three-dimensional measurement task. The three-dimensional measurement method according to claim 21, wherein the determining of the multiple algorithm function modules required to perform the three-dimensional measurement task and the connection relationship between the multiple algorithm function modules comprises: Determine a plurality of algorithm function modules corresponding to newly added functions of the host program, wherein the newly added functions are used to process the three-dimensional measurement task; The process of driving and executing the three-dimensional measurement task includes: Using the host program to load multiple plug-ins corresponding to the multiple algorithm function modules, and importing the multiple plug-ins into the host program; The host program into which the plurality of plug-ins are imported is called to drive and execute the process of the three-dimensional measurement task. The three-dimensional measurement method according to claim 21, further comprising: In response to the first operation, determining a hardware access module required for the three-dimensional measurement task and a connection relationship between the hardware access module and the plurality of algorithm function modules, wherein different hardware access modules are used to access different hardware devices; The plug-in corresponding to the hardware access module is parsed to record the connection relationship between the hardware access module and the multiple algorithm function modules. An information processing method, comprising: In response to a first operation of a user in the flowchart, a plurality of algorithm function modules required to perform an information processing task and a connection relationship between the plurality of algorithm function modules are determined to generate a flow of the information processing task, wherein the algorithm function modules are packaged in a plug-in form; Parsing multiple plug-ins corresponding to the multiple algorithm function modules to record the connection relationship between the multiple algorithm function modules; According to the recorded connection relationship, the process of the information processing task is driven and executed. An electronic device, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the image processing method described in any one of claims 1-20, or the three-dimensional measurement method described in any one of claims 21-24, or the information processing method described in claim 25 based on instructions stored in the memory device. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 20, or the three-dimensional measurement method according to any one of claims 21 to 24, or the information processing method according to claim 25. A computer program comprising: Instructions, when executed by a processor, cause the processor to perform the image processing method according to any one of claims 1 to 20, or the three-dimensional measurement method according to any one of claims 21 to 24, or the information processing method according to claim 25.

Images