CN114996136A - Test data determination method and device, electronic equipment and readable storage medium - Google Patents

Abstract

The disclosure provides a test data determination method and device, an electronic device and a readable storage medium, wherein the method comprises the following steps: carrying out neural network search in a preset search space, and determining to-be-tested data according to a search result; running the data to be tested through a neural network processing system to obtain a running result of the data to be tested; and under the condition that the operation result is abnormal operation, determining the data to be tested as target test data, wherein the target test data is used for improving the neural network processing system. According to the embodiment of the disclosure, the test efficiency and the test coverage rate of the neural network processing system can be improved.

Description

Test data determination method and device, electronic device, and readable storage medium

technical field

The present disclosure relates to the technical field of testing, and in particular, to a method and apparatus for determining test data, an electronic device, and a readable storage medium.

Background technique

Due to the high design complexity of the neural network processing system, the testing complexity of the neural network processing system is increased. Developers often need to find test cases in a large number of neural networks that will make the entire neural network processing system fail or deteriorate in performance, so as to improve the design of the neural network processing system.

In the usual testing methods, testers are often used to manually construct test cases, such as a test network for testing neural network systems, or manually write rules to generate test cases to test the entire neural network test system, but by manually constructing tests Use cases, or tests that manually write rules are more difficult and less efficient.

SUMMARY OF THE INVENTION

The present disclosure provides a test data determination method and device, an electronic device, and a readable storage medium.

In a first aspect, the present disclosure provides a method for determining test data, the method for determining test data includes: performing a neural network search in a predetermined search space, and determining data to be tested according to the search results; running the data to be tested through a neural network processing system, The running result of the data to be tested is obtained; if the running result is abnormal, the data to be tested is determined as the target test data, and the target test data is used to improve the neural network processing system.

In a second aspect, the present disclosure provides a test data determination device, the test data determination device includes: a search module for performing a network structure search in a predetermined neural network search space, and determining the data to be tested according to the search result; an operation module , which is used to run the data to be tested through the neural network processing system to obtain the running result of the data to be tested; the determination module is used to determine the data to be tested as the target test data when the running result is abnormal. for improving neural network processing systems.

In a third aspect, the present disclosure provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data that can be processed by the at least one processor One or more computer programs executed by the processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described test data determination method.

In a fourth aspect, the present disclosure provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned test data determination method when executed by a processor/processing core.

The embodiments provided by the present disclosure can perform a neural network search from a predetermined search space to obtain target test data that causes the neural network processing system to operate abnormally, so as to use the target test data that causes the neural network processing system to operate abnormally to improve neural network processing system, which is conducive to improving test efficiency and test coverage.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification, and together with the embodiments of the present disclosure, they are used to explain the present disclosure, and are not intended to limit the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

1 is a flowchart of a method for determining test data according to an embodiment of the present disclosure;

2 is a schematic diagram of a system framework for neural network architecture search provided by an embodiment of the present disclosure;

3 is a schematic flowchart of a method for determining test data provided by an exemplary embodiment of the present disclosure;

4 is a block diagram of an apparatus for determining test data according to an embodiment of the present disclosure;

FIG. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

In order for those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, and they should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

Various embodiments of the present disclosure and various features of the embodiments may be combined with each other without conflict.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used herein is used to describe particular embodiments only and is not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that when the terms "comprising" and/or "made of" are used in this specification, the presence of features, integers, steps, operations, elements and/or components is specified, but does not preclude the presence or addition of a or multiple other features, integers, steps, operations, elements, components and/or groups thereof. Words like "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be construed as having meanings consistent with their meanings in the context of the related art and the present disclosure, and will not be construed as having idealized or over-formal meanings, unless expressly so limited herein.

Due to the emergence of more and more neural network acceleration systems (such as neural network accelerators) in related technologies, the design of accelerators has become more and more complex. Many neural network acceleration systems are no longer only for single neural network acceleration, but more general support Neural networks with different structures and operators. This increases the design complexity of the entire system (compiler and hardware devices), which in turn increases the testing complexity. Developers often need to find target test data that will make the entire system fail or deteriorate in performance in countless neural networks, so as to improve the design of the entire neural network acceleration system. The test method for the neural network acceleration system becomes particularly important.

In practical application scenarios, when faced with complex neural network processing systems, the usual test methods are often difficult to achieve high test coverage, especially with the continuous development of the current many-core architecture, which makes the network structure, Complex balance, etc. have become important aspects that affect execution performance. The combination of these complex rules makes it difficult to manually construct test cases; however, it is difficult to test the entire system by manually constructing test cases or manually writing rules to generate test cases. high and the test is inefficient.

According to the test data determination method of the embodiment of the present disclosure, target test data that will cause abnormal operation of the neural network processing system can be generated through network structure search, so that the entire neural network processing system can be improved through the target test data, and the test of the neural network processing system can be improved. Efficiency, making the target test data used to test the neural network processing system more sufficient and complete, thereby improving the test coverage.

The following first describes the basic flow of neural network search through Figure 1. FIG. 1 shows a schematic diagram of a system framework for neural network search provided by an embodiment of the present disclosure.

In FIG. 1 , the framework includes: a search space acquisition module 101 , a network search module 102 and a model evaluation module 103 . The basic flow of neural network search through this system framework can include the following steps.

S101, a search space is predefined by the search space obtaining module 101.

In this step, the search space can be understood as a candidate set of network structures to be searched, that is, a rough and initial neural network framework. The search space can be used to define the type of neural network searched by the neural network search algorithm, as well as how to describe the neural network structure.

S102 , the network search module 102 searches out one or more neural networks from the search space, and sends the searched neural networks to the model evaluation module 103 .

S103 , the model evaluation module 103 generates a performance evaluation result of the searched neural network by evaluating the performance of the searched neural network when dealing with deep learning problems, and returns the performance evaluation result to the network search module 102 .

In this step, deep learning (DL) is a method in machine learning (ML) for learning the inherent laws and representation levels of data; processing deep learning problems through neural networks can be understood as using The deep learning method imitates the analysis and learning ability of the human brain by establishing a neural network, and solves more complex pattern recognition problems, such as recognizing data such as speech, image and text; using performance evaluation indicators can be used to measure the performance of neural networks in processing. The performance of deep learning problems, the performance of neural networks in dealing with deep learning problems is closely related to the structural advantages and disadvantages of neural networks. Therefore, performance evaluation indicators can be used to measure the structural advantages and disadvantages of neural networks.

In this step, a predetermined search strategy can be used to search out the neural network structure from the search space; and a performance evaluation index can be used to measure the structure of the neural network; the performance evaluation index includes at least one of the following index items: accuracy, precision , speed, delay, throughput, computing occupancy, storage occupancy, parallelism, power consumption, etc.

S104, the network search module 102 constructs a loss function according to the performance evaluation result (for example, the performance value obtained from the evaluation), so as to optimize the network structure of the neural network search in the network search module 102 through the loss function, and iterate in this way to generate a better evaluation result. Network structure for doing neural network searches.

It can be known from the basic flow of the neural network architecture search described in FIG. 1 that the neural network can be obtained from the search space, and the neural network obtained by the search can be used as the evaluated neural network. The embodiments of the present disclosure can perform neural network architecture search based on the NAS network, perform neural network search and performance evaluation, so as to generate a neural network that will cause abnormal operation of the neural network processing system.

FIG. 2 is a flowchart of a method for determining test data according to an embodiment of the present disclosure. 2, the method includes the following steps.

S210: Perform a neural network search in a predetermined search space, and determine the data to be tested according to the search result.

S220 , running the data to be tested through the neural network processing system to obtain a running result of the data to be tested.

S230, in the case that the operation result is abnormal operation, determine the data to be tested as test data, and the target test data is used to improve the neural network processing system.

According to the test data determination method of the embodiment of the present disclosure, it is possible to perform a neural network search in a predetermined search space to obtain the data to be tested that causes the neural network processing system to operate abnormally, so that the neural network can be improved by using the network that causes the neural network processing system to operate abnormally. Process systems to improve test efficiency and test coverage.

In some embodiments, the step of performing a neural network search in a predetermined search space in step S210 may be performed by a search component; the search component may be implemented as a neural network; when the search component is a neural network, the neural network may provide The function of performing neural network search in a given search space; the search component can also be implemented as: a component (such as a searcher) that invokes an external search engine or external search tool to perform neural network search, etc.; in the embodiment of the present disclosure, When the search component is called by the electronic device that executes the test data determination method, the operation of performing a neural network search in a predetermined search space is performed.

In this embodiment of the present disclosure, a search result obtained by performing a neural network search in a predetermined search space includes: at least one of a neural network model, a computational graph model, and a program use case related to a network structure; the test of the embodiment of the present disclosure In the data determination method, at least one of a neural network model, a computational graph model, and a program use case related to the network structure included in the search result can be used as the data to be tested. That is, in some embodiments, the data to be tested may include at least one of the following items: at least one of a neural network model, a computational graph model, and a program use case related to a network structure.

The search space in the embodiment of the present disclosure is a search space defined in advance according to the operator library of the neural network processing system and the test requirements; the operator library is used to indicate the set of operator types supported by the neural network processing system; the test requirements are at least It includes one of the following information items: the network scale required for this system test, the network connection constraints for this system test, and the operator type obtained from the operator type set that needs to be included in this system test.

In this embodiment, the operator library and the test requirements can define a search space, and the neural network search is performed from the search space, and finally the target test data that makes the neural network processing system run abnormally is obtained by searching, and the obtained target test data is used. Improve the neural network processing system to improve test efficiency and test coverage.

In some embodiments, the search component includes at least one of the following neural networks: a recurrent neural network and an evolutionary algorithm-based neural network.

In the embodiment of the present disclosure, when the search component is a neural network, a cyclic neural network may be used, or a neural network with a general search algorithm such as an evolutionary algorithm may be used. The embodiment of the present disclosure does not limit the selection of the neural network used by the search component. Specifically, the network type of the neural network used may be determined according to actual needs, which is not specifically limited in the embodiment of the present disclosure.

In some embodiments, the neural network processing system includes a neural network acceleration system including a neural network deployment device and a neural network processing device; in some embodiments, the neural network deployment device may be a compiler, an interpreter, and a At least one of the mappers; the neural network processing device may include hardware devices such as a neural network chip or a neural network board; exemplarily, the neural network chip may include but is not limited to: many-core chips, neural network processors (Neural At least one of Network Processor, NNP) and Deep Learning Processor (Data Learning Processing Unit, DPU).

According to the test data determination method of the embodiment of the present disclosure, the target test data that causes the neural network processing system to run abnormally can be searched from a predetermined search space, thereby improving the design of the neural network acceleration system, and obtaining the abnormal operation of the neural network processing system. The difficulty of the target test data is low, which is beneficial to improve the test efficiency and test coverage.

The test data determination method in the embodiment of the present disclosure may be performed by an electronic device such as a terminal device or a server, and the terminal device may be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, For personal digital assistants (Personal Digital Assistants, PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by the processor calling the computer-readable program instructions stored in the memory. Alternatively, the method can be performed by a server.

In some embodiments, the abnormal operation includes: the operation result is that the operation fails; or, the operation result is that the operation is successful, and the operation index value in the operation result is less than a predetermined operation index threshold.

In this embodiment, the test data that causes the neural network processing system to fail or have poor performance can be searched from the predetermined search space, thereby improving the design of the neural network acceleration system. Difficulty with poor test data and improve test efficiency and test coverage.

In some embodiments, the operation index value includes at least one of a system performance index value, a system pressure index value, a system power consumption index value, and a system security index value.

As an example, the system performance index value is used to evaluate system performance, and the system performance index may include, but is not limited to, at least one of the following index items: response time, CPU resource utilization, and memory utilization.

As an example, the system pressure index value is used to evaluate the maximum load that the system can support under normal use; the system pressure index value may include, but is not limited to, at least one of the following index items: Transaction Per Second (TPS), Average response time, maximum response time, minimum response time, etc. Among them, the number of transactions processed per second is the number of transactions processed by the system per second, which can be used to evaluate the time transaction load of the system at any given moment; the average response time is the average time taken for each transaction to execute, which can be used to evaluate the test scenario. The performance trend of the system during operation; the maximum response time refers to the maximum time from issuing an instruction to the system responding; the minimum response time refers to the minimum time from issuing an instruction to the system responding.

As an example, the system power consumption indicator value is used to evaluate the amount of energy consumed per unit time of the system. The system power consumption index value may include, but is not limited to, at least one of the following index items: power consumption and standby time.

As an example, the system security index value is used to evaluate the system security and the degree of data protection; as an example, the system security index value may include, but is not limited to, at least one of the following index items: confidentiality index (data encryption correctness , encryption algorithm strength, etc.), integrity indicators (such as data integrity).

It should be understood that the above-mentioned system performance index value, system pressure index value, system pressure index value, and system power consumption index value may also include index values of other corresponding types of index items, and the above-mentioned specific index items in the embodiments of the present disclosure are only illustrative. illustrate. In practical application scenarios, custom settings can be made according to actual needs.

In this embodiment, different types of system tests can be performed on the neural network processing system through different types of operating index values, such as at least one of performance tests, system stress tests, system power consumption tests, and system security tests; The selection of the operation index value may be determined according to the actual test requirements for the neural network processing system, which is not specifically limited in the embodiment of the present disclosure.

In some embodiments, in step S230, if the operation result is abnormal operation, determine the data to be tested as the target test data, which may specifically include: S11, if the operation result is that the operation fails, determine the data to be tested as the target test Error-type test data in the data is used for system debugging of the neural network processing system; S12, if the operation result is successful, and the operation index value in the operation result is less than the operation index threshold, determine the data to be tested as the target test Degraded test data in data for system optimization of neural network processing systems.

In this embodiment, if the operation result is abnormal operation, the error-type test data (which may be referred to as an error test case) can be obtained; if the operation result is that the operation is successful but the operation index value is less than the operation index threshold, it can be Obtain performance-reducing test data (which can be called performance-reducing test cases); both error test cases and performance-reducing test cases can be used as target test cases for abnormal operation of the neural network processing system to optimize the neural network processing system. , reduces the difficulty of obtaining neural networks that make the system behave abnormally, and improves test efficiency and test coverage.

In some embodiments, the network structure search is based on a search component; the search component has a neural network structure; the test data determination method further includes: S21, when the operation result is that the operation is successful, and the operation index value in the operation result is greater than or equal to the predetermined operating index threshold, construct a loss function of the search component according to the operating index value; S22, update the network parameters of the search component according to the loss function.

In some embodiments, the step of constructing the loss function of the search component according to the operation index value in the above step S21 may specifically include: S31, determining the loss function value according to the operation index value and the corresponding type of operation index threshold; S32, increasing the loss The function value is to construct the loss function of the search component according to the added loss function value; or, it may specifically include: constructing the loss function of the search component according to the inverse of the specified performance index value, and the specified performance index value is greater than or equal to the running index value.

In the embodiment of the present disclosure, a loss function (Loss Function) may be used to estimate the gap between the running result of the search component and the running target, that is, the running error of the search component. For example, the operation result of the search component is the operation index value obtained each time the search component runs, and the operation target of the search component is that the search component runs in error or the operation index value is smaller than the operation index threshold of the corresponding type. By increasing the loss function value, the probability of successful operation of the search component can be reduced, or the probability of the operation result of the search component being close to or reaching the operating index threshold can be reduced, which is beneficial to improve the probability of obtaining an error-like neural network or a neural network with reduced performance. probability.

In this embodiment, using a larger loss function value or the opposite of a higher performance index value can promote the NAS network to search to obtain more performance-decreasing neural networks or more error-like neural networks, and speed up the acquisition of better neural networks. Handle abnormal neural networks in system operation, improve test efficiency and test coverage.

According to the test data determination method of the embodiment of the present disclosure, a neural network search can be performed in a predetermined search space through a NAS network to obtain a network that causes the neural network processing system to run abnormally, so that the neural network processing system can be used to improve the neural network abnormally. The network processing system is beneficial to reduce the difficulty of obtaining target test data that makes the neural network processing system run abnormally, and improve the test efficiency and test coverage.

FIG. 3 shows a schematic flowchart of a test data determination method provided by an exemplary embodiment of the present disclosure. In Figure 3, the test data determination method may include the following steps:

S301, a search space is defined according to an operator library of a neural network processing system and a test requirement.

In this step, the operator library includes all operator types supported by the neural network processing system, and the test requirements may include the operator types, network scale, network connection constraints, etc. that need to be included in this test.

S302: Search the neural network from the search space through the search component, and determine the data to be tested according to the search result.

In this step, the search component has a neural network structure, and uses at least one of a neural network model, a computational graph model, and a program use case related to the network structure in the search result as the data to be tested.

S303: Input the searched data to be tested into the neural network deployment device, process the searched neural network through the neural network deployment device, and obtain a neural network processed by the neural network deployment device.

As an example, when the neural network deployment device is a compiler, the searched neural network is processed by the neural network deployment device, and the neural network processed by the neural network deployment device is obtained, including: compiling the searched neural network through the compiler to obtain The compiled neural network.

S304 , running the neural network processed by the neural network deployment device through the neural network chip to obtain the running result.

S305, it is judged whether there is an error in the operation of the system, if there is an error in the operation, step S306 is performed, and if the operation is successful, step S307 is performed.

In this step, the operation error may include, for example, at least one of: operation interruption, system crash, and interface confusion; the operation error may also include other problems with hardware or system software, which can be customized by the user according to actual test requirements.

S306 , in the case of an error in the operation, the data to be tested is used as error-type test data as an error test case.

S307, in the case of successful operation, determine whether the actual operation performance value of the neural network processing system is lower than the predefined expected operation performance threshold; if the actual operation performance value is lower than the expected operation performance threshold, execute step S308; otherwise, Step S309 is executed.

S308 , in the case that the actual operating performance value is lower than the expected operating performance threshold, use the data to be tested as performance-reducing type test data, that is, obtain a performance-reducing test case.

In this step, the running performance may be predefined according to the performance index specified in the requirement specification, and may also be user-defined according to the test requirement, which is not specifically limited in the embodiment of the present disclosure.

S309, when the actual operating performance value is greater than or equal to the expected operating performance threshold, calculate the loss function of the neural network structure of the search component, use the loss function to train the neural network structure, and obtain an updated search component; and execute step S302.

In this step, the inverse number of the actual operating performance value can be used, or the loss function value can be calculated according to the actual operating index value and the expected operating performance threshold value, and the loss function value can be increased, and the search component can be constructed according to the increased loss function value. The loss function of the neural network structure; in this step, the calculation method of the loss function satisfies that the higher the actual running performance, the larger the loss function value, so as to speed up the speed of obtaining the target test data and improve the test efficiency.

In this embodiment of the present invention, after step S309, the above steps S302-S306 are iteratively executed, so that the search component searches out the target test data from the search space that makes the neural network processing system perform worse, so that the neural network processing system has poorer performance. Improvements are made to improve the test efficiency and test coverage of the neural network processing system.

In some embodiments, in the iterative execution process, until the test end condition is satisfied, it is determined that the test ends, and the target test data that causes the neural network processing system to run incorrectly and/or degrade the running performance is obtained. The test end condition may include: the quantity of target test data obtained according to the running result for N consecutive times is zero, or the quantity of obtained target test data reaches the threshold of the quantity of target test data. For example, the amount of error-type test data is greater than or equal to the threshold for the number of error-type test data, and/or the amount of performance-degraded test data is greater than or equal to the number of performance-degraded test data thresholds.

According to the test data determination method of the embodiment of the present invention, a neural network search can be performed in a predetermined search space to obtain target test data that causes the neural network processing system to operate abnormally, so that the target test data that causes the neural network processing system to operate abnormally can be used to Testing and improving the neural network processing system is beneficial to reduce the difficulty of obtaining target test data that makes the neural network processing system run abnormally, and improve the test efficiency and test coverage.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides a test data determination device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any test data determination method provided by the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the method section. ,No longer.

FIG. 4 is a block diagram of an apparatus for determining test data according to an embodiment of the present disclosure.

Referring to FIG. 4 , an embodiment of the present disclosure provides an apparatus 400 for determining test data. The apparatus 400 for determining test data includes the following modules.

The search module 410 is configured to perform a neural network search in a predetermined search space, and determine the data to be tested according to the search result.

The running module 420 is used for running the data to be tested through the neural network processing system to obtain the running result of the data to be tested.

The determination module 430 is configured to determine the data to be tested as target test data when the operation result is abnormal operation, and the target test data is used to improve the neural network processing system.

In some embodiments, the abnormal operation includes: the operation result is that the operation fails; or, the operation result is that the operation is successful, and the operation index value in the operation result is less than a predetermined operation index threshold.

In some embodiments, when the operation result is abnormal operation, when the determination module 430 is used to determine the data to be tested as the target test data, the determination module 430 may specifically include: an error type test data determination unit, configured to determine if the operation result is If the operation fails, the data to be tested is determined to be the error-type test data in the target test data, which is used for system debugging of the neural network processing system; the performance-reduced type test data determination unit is used if the operation result is successful and the operation If the operation index value in the result is less than the operation index threshold, the data to be tested is determined to be the performance-degraded test data in the target test data, so as to be used for system optimization of the neural network processing system.

In some embodiments, the neural network search performed in the predetermined search space is based on a search component; the search component has a neural network structure; the test data determination apparatus 400 further includes: a loss function building module for running the result as When the operation is successful and the operation index value in the operation result is greater than or equal to the predetermined operation index threshold, the loss function of the search component is constructed according to the operation index value; the parameter update module is used to update the network parameters of the search component according to the loss function .

In some embodiments, the loss function building module, when used to construct the loss function of the NAS network according to the operation index value, can be specifically used to determine the loss function value according to the operation index value and the corresponding type of operation index threshold; increase the loss function value, the loss function of the search component is constructed according to the increased loss function value; or, it can be used to construct the loss function of the search component according to the inverse of the specified performance index value, and the specified performance index value is greater than or equal to the running index value.

In some embodiments, the operation index value includes at least one of a system performance index value, a system pressure index value, a system power consumption index value, and a system security index value; the search component includes at least one of the following neural networks: Recurrent Neural Networks and Evolutionary Algorithm-based Neural Networks.

In some embodiments, the neural network deployment device includes: at least one of a compiler, an interpreter, and a mapper; the neural network processing device includes: at least one of a neural network chip and a neural network board.

In some embodiments, the data to be tested includes at least one of the following items: at least one of a neural network model, a computational graph model, and a program use case related to a network structure.

In some embodiments, the neural network processing system includes a neural network acceleration system, and the neural network acceleration system includes a neural network deployment device and a neural network processing device; the search space is defined in advance according to the operator library and test requirements of the neural network processing system The operator library is used to indicate the set of operator types supported by the neural network processing system; the test requirements include at least one of the following information items: the network scale required for this system test, the network connection for this system test Constraints, as well as the operator types obtained from the operator type set that need to be included in this system test.

According to the test data determination device of the embodiment of the present invention, the target test data that causes the neural network processing system to operate abnormally can be obtained by performing a neural network search in a predetermined search space, so that the target test data that causes the neural network processing system to operate abnormally can be used to improve the neural network. The network processing system is beneficial to reduce the difficulty of obtaining target test data that makes the neural network processing system run abnormally, and improve the test efficiency and test coverage.

FIG. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.

5, an embodiment of the present disclosure provides an electronic device, the electronic device includes: at least one processor 501; and a memory 502 connected in communication with the at least one processor 501; One or more computer programs executed by the processor 501, the one or more computer programs are executed by the at least one processor 501, so that the at least one processor 501 can execute the above-mentioned test data determination method.

Embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned test data determination method when executed by a processor/processing core. Computer-readable storage media can be volatile or non-volatile computer-readable storage media.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are executed in a processor of an electronic device , the processor in the electronic device executes the above test data determination method.

Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, functional modules/units in the systems, and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components Components execute cooperatively. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable storage media, which may include computer storage media (or non-transitory media) and communication media (or transitory media).

As is known to those of ordinary skill in the art, the term computer storage media includes both volatile and non-volatile memory media implemented in any method or technology for storage of information, such as computer readable program instructions, data structures, program modules or other data. volatile, removable and non-removable media. Computer storage media include, but are not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, Portable compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or can be used to store desired information and can be accessed by a computer any other medium. In addition, communication media typically embodies computer readable program instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery, as is well known to those of ordinary skill in the art medium.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code written in any combination, programming languages including object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present disclosure.

The computer program product described herein may be embodied in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executables for implementing the specified logical function(s) instruction. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should only be construed in a general descriptive sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that features, characteristics and/or elements described in connection with a particular embodiment may be used alone or in combination with other embodiments, unless expressly stated otherwise. Features and/or elements are used in combination. Accordingly, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure as set forth in the appended claims.

Claims (12)

1. A method for determining test data, comprising:

carrying out neural network search in a preset search space, and determining data to be tested according to a search result;

running the data to be tested through a neural network processing system to obtain a running result of the data to be tested;

and under the condition that the operation result is abnormal operation, determining the data to be tested as target test data, wherein the target test data is used for improving the neural network processing system.

2. The method of claim 1, wherein the neural network search conducted in the predetermined search space is a search conducted based on a search component; the search component has a neural network structure; the method further comprises the following steps:

under the condition that the operation result is successful and the operation index value in the operation result is greater than or equal to a preset operation index threshold value, constructing a loss function of the search component according to the operation index value;

and updating the network parameters of the search component according to the loss function.

3. The method of claim 2, wherein constructing a loss function for the search component from the operational metric values comprises:

determining a loss function value according to the operation index value and an operation index threshold value of a corresponding type;

increasing the loss function value, and constructing a loss function of the search component according to the increased loss function value;

or constructing a loss function of the search component according to the inverse of a specified performance index value, wherein the specified performance index value is greater than or equal to the running index value.

4. The method of claim 1, wherein,

the neural network processing system comprises a neural network acceleration system, and the neural network acceleration system comprises a neural network deployment device and a neural network processing device;

the preset search space is defined in advance according to an operator library of the neural network processing system and test requirements;

the operator library is used for indicating the operator type set supported by the neural network processing system;

the test requirement comprises at least one of the following information items: the network scale required by the system test, the network connection constraint of the system test, and the operator type required by the system test, which is acquired from the operator type set.

5. The method of claim 1, wherein the operational anomaly comprises:

the operation result is operation failure; or,

and the operation result is successful operation, and the operation index value in the operation result is smaller than a preset operation index threshold value.

6. The method of claim 5, wherein in the case that the operation result is an operation abnormality, determining the data to be tested as target test data comprises:

if the operation result is operation failure, determining that the data to be tested is error test data in the target test data, so as to carry out system debugging on the neural network processing system;

and if the operation result is successful and the operation index value in the operation result is smaller than the operation index threshold, determining the data to be tested as performance reduction type test data in the target test data for carrying out system optimization on the neural network processing system.

7. The method of claim 2, wherein,

the operation index value includes: at least one of a system performance index value, a system pressure index value, a system power consumption index value, and a system safety index value;

the search component includes at least one of the following neural networks: recurrent neural networks and neural networks based on evolutionary algorithms.

8. The method of claim 4, wherein,

the neural network deployment device includes: at least one of a compiler, an interpreter, and a mapper;

the neural network processing device includes: at least one of a neural network chip and a neural network board card.

9. The method of any one of claims 1-8,

the data to be tested comprises at least one of: at least one of a neural network model, a computational graph model, and a program use case related to a network structure.

10. A test data determination apparatus comprising:

the searching module is used for searching the neural network in a preset searching space and determining data to be tested according to a searching result;

the operation module is used for operating the data to be tested through a neural network processing system to obtain an operation result of the data to be tested;

and the determining module is used for determining the data to be tested as target test data under the condition that the operation result is abnormal operation, wherein the target test data is used for improving the neural network processing system.

11. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores one or more computer programs executable by the at least one processor to enable the at least one processor to perform the test data determination method of any one of claims 1-9.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the test data determination method according to any one of claims 1 to 9.

Images