WO2025069198A1 - Setting assistance device, setting assistance method, and setting assistance program - Google Patents

Abstract

When the inference result of a first inference with a light processing load satisfies a threshold condition, a setting assistance device (100) varies a threshold value used for two-layer inference for performing a second inference with a processing load heavier than that of the light inference, and collates, with correct answer data, the result of two-layer inference for each varied threshold value, the result being obtained by performing two-layer inference for each varied threshold value. The setting assistance device (100) then creates a correlation graph between the varied threshold value and the accuracy and performance of two-layer inference for each varied threshold value, the accuracy and performance being calculated as the result of collation.

Description

Setting assistance device, setting assistance method, and setting assistance program

The present invention relates to a setting assistance device, a setting assistance method, and a setting assistance program.

In recent years, advances in deep learning and the spread of IoT (Internet of Things) devices have led to the development of applications for automatic analysis of surveillance camera footage, autonomous driving, and the like. These applications perform prescribed processing based on the results of analyzing video and images, so real-time performance is important. Therefore, to ensure real-time performance, it is desirable for DNN (Deep Neural Network) inference, etc. to be performed on edge devices such as IoT devices.

However, because edge devices have significant resource constraints, it is difficult to keep a highly accurate DNN model resident and running. Therefore, in order to achieve both accuracy and real-time performance, a framework has been proposed in which low-accuracy but low-load inference (hereinafter sometimes referred to as "light inference") is performed on the edge device, and high-accuracy inference (hereinafter sometimes referred to as "heavy inference") is performed on the cloud side when it is determined to be necessary. For example, when performing DNN inference, a technology is known in which the necessity of the next inference is determined based on the output results such as the confidence level from the light inference and a threshold value, and heavy inference processing is performed when it is determined to be necessary, thereby reducing the frequency of heavy inference and improving the performance of the entire system (for example, see Non-Patent Document 1).

NoScope: Optimizing Neural Network Queries over Video at Scale <URL: https://www.vldb.org/pvldb/vol10/p1586-kang.pdf> <Retrieved September 25, 2023>

However, the above-mentioned conventional technology has an issue in that it is not easy to tune the threshold value used for inference. For example, the threshold value used to determine whether or not to perform heavy inference is a delicate value that affects accuracy and performance, so tuning is required to achieve the required accuracy and performance depending on the inference use case. However, the process of verifying each threshold that balances accuracy and performance and determining the optimal threshold is a heavy load.

In order to solve the above-mentioned problems and achieve the object, the setting assistance device of the present invention is characterized by having a matching unit that varies the threshold value used in a second-layer inference in which a second inference with a heavier processing load than a first inference is performed when the inference result of a first inference with a lighter processing load satisfies a threshold condition, and performs the second-layer inference for each varied threshold value, and compares the results of the second-layer inference for each varied threshold value obtained by performing the second-layer inference for each varied threshold value with correct answer data, and a creation unit that creates a correlation graph between the varied threshold value and the accuracy and performance of the second-layer inference for each varied threshold value calculated as the comparison result by the matching unit.

The present invention has the effect of making it easier to tune the thresholds used in inference.

FIG. 1 is a diagram for explaining an overall picture of the processing of the setting assistance device according to the present embodiment. FIG. 2 is a diagram showing an example of a process performed by the setting assistant device according to the present embodiment. FIG. 3 is a diagram showing an example of the configuration of the setting assistance device according to the present embodiment. FIG. 4 is a table showing an example of a matching result according to the present embodiment. FIG. 5 is a table diagram showing an example of an inference condition according to this embodiment. FIG. 6 is a diagram showing an example of auxiliary setting information to be output according to the present embodiment. FIG. 7 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 8 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 9 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 10 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 11 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 12 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 13 is a diagram showing an example of generation of auxiliary setting information according to the present embodiment. FIG. 14 is a flowchart showing an example of a procedure for generating auxiliary setting information according to the present embodiment. FIG. 15 is a diagram illustrating an example of a computer that executes a setting assistance program according to this embodiment.

Below, the form for carrying out the present invention (hereinafter, "embodiments") will be described with reference to the drawings. Note that each embodiment is not limited to the contents described below.

<Overview of processing by the setting assistant device>
Fig. 1 is a diagram for explaining an overall picture of the processing of the setting assistance device 100 according to this embodiment. The setting assistance device 100 shown in Fig. 1 is an example of a computer that provides a technology for repeatedly performing two-layer inference while changing a threshold value (hereinafter, may be simply referred to as "threshold value") used to determine whether or not to perform heavy inference in two-layer inference, and for comparing the inference result of the two-layer inference with correct answer data to create a correlation graph between the threshold value, accuracy, and performance.

As mentioned above, it is desirable to perform DNN inference etc. on the edge of IoT devices to ensure real-time performance. However, due to resource constraints etc., it is difficult to keep a highly accurate DNN model running on edge devices.

Therefore, a reference technology is known that performs inference in two stages (two-layer inference) to reduce the frequency of heavy inference, which has a high processing load, and improve the performance of the entire two-layer inference system. For example, as shown in FIG. 1, a two-layer inference system 10 outputs predetermined numerical values such as difference and confidence levels by performing light inference on the edge side using a lightweight model with a low processing load, such as a small amount of calculation, and compares the results with a preset threshold value. Then, the two-layer inference system 10 performs heavy inference processing on the cloud side only if it is determined as a result of the comparison that heavy inference (high-precision DNN inference) should be performed.

The thresholds used in the two-layer inference system 10 described above are delicate values that affect accuracy and performance, and therefore require tuning to suit the purpose of the inference, but there is a problem in that such tuning is not easy. For example, the performance required for two-layer inference is defined by the application of the user who uses the two-layer inference. The thresholds are then determined to satisfy the required performance described above, but there is a trade-off between accuracy and performance that varies depending on the threshold settings, such as when emphasis is placed on performance, the accuracy decreases. For this reason, users who use two-layer inference need to tune the thresholds according to the accuracy and performance required for each use case of the two-layer inference. However, when tuning, it is not easy to verify the optimal threshold by comparing the balance between accuracy and performance for each threshold, which is a lot of work.

The setting assistance device 100 according to this embodiment varies the threshold used in the second-layer inference, which performs an inference (second inference) that has a heavier processing load than the light inference, when the inference result of the inference (first inference) with a light processing load satisfies the threshold condition, and performs the second-layer inference for each varied threshold, and compares the results of the second-layer inference for each varied threshold with the correct answer data.The setting assistance device 100 then creates a correlation graph between the varied threshold, the accuracy for each varied threshold calculated as the comparison result, and the performance of the second-layer inference.

Specifically, as shown in FIG. 1 (1), the setting assistance device 100 performs two-layer inference using moving images, still images, etc. as input data. At this time, the setting assistance device 100 repeatedly performs two-layer inference while varying the threshold value used for judgment based on predetermined conditions, and outputs the result of the two-layer inference for each varied threshold value.

As shown in (2) of FIG. 1, the configuration assistance device 100 compares the output inference results of the two-layer inference with correct answer data such as data generated by heavy inference using a highly accurate second-layer model or data generated in advance manually (e.g., ground truth, etc.).

As shown in FIG. 1 (3), the configuration assistance device 100 creates a correlation graph or the like that associates predetermined indices that indicate the accuracy calculated by matching the inference results of the two-layer inference with the correct answer data and the performance calculated from the results of the two-layer inference with each of the varied thresholds.

In this way, the setting assistance device 100 automatically creates a correlation graph showing the relationship between accuracy, performance, and thresholds based on the results of matching the results of two-layer inference with ground truth data, making it easy to tune thresholds to suit the purpose and needs of the inference.

<Explanation of setting assistance device>
From here, the setting assistance device 100 will be described. First, the processing by the setting assistance device 100 according to this embodiment will be described. FIG. 2 is a diagram showing an example of processing by the setting assistance device 100 according to this embodiment. FIG. 2 shows an example in which the setting assistance device 100 outputs setting assistance information (hereinafter, sometimes simply referred to as "setting assistance information") such as a correlation graph that associates the changed threshold with the accuracy and performance of the two-layer inference based on the comparison result between the result of the two-layer inference for each threshold and the correct answer data, a recommended threshold, and a comparison result of the change in the inference result, and sets the thresholds and the like of the two-layer inference system 10 using the output setting assistance information.

First, as shown in FIG. 2 (1), the configuration assistance device 100 accepts specifications of input data to be input to the model used for two-layer inference, models to be used for light and heavy inference in two-layer inference, correct answer data to be used in matching processing, inference conditions for two-layer inference, etc.

As shown in FIG. 2 (2), the setting assistance device 100 inputs specified input data for the model used in the two-layer inference, and performs light inference while varying the thresholds used for judgment based on the specified inference conditions. The setting assistance device 100 then performs heavy inference and outputs the results obtained from the heavy inference as the result of the two-layer inference when the degree of difference, certainty, etc. obtained as a result of the light inference satisfy the judgment conditions for each of the varied thresholds. The setting assistance device 100 repeatedly performs the above-mentioned process according to the specified inference conditions.

On the other hand, as shown in (3) of FIG. 2, the configuration assistance device 100 performs heavy inference generation of correct answer data to be used in the matching process and accepts data generated in advance (ground truth), etc., in order to prepare correct answer data.

As shown in (4) of FIG. 2, the setting assistance device 100 compares the inference results of the two-layer inference with the correct answer data. Then, as shown in (5) of FIG. 2, the setting assistance device 100 creates setting assistance information such as a correlation graph that associates accuracy and performance for each threshold value, and a comparison result of recommended threshold values and changes in inference results, based on the comparison results.

As shown in FIG. 2 (6), the setting assistance device 100 outputs setting assistance information to the user, such as a correlation graph that associates the created accuracy with the performance of the two-layer inference, recommended thresholds, and comparison results of changes in inference results. Then, as shown in FIG. 2 (7), the user uses the output setting assistance information to set thresholds, etc., for the two-layer inference system 10, etc.

(Setting assistant device 100)
Next, the configuration of the setting assistance device 100 and the functions of each functional unit will be described. FIG. 3 is a diagram showing an example of the configuration of the setting assistance device 100 according to this embodiment. As shown in FIG. 3, the setting assistance device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Although not shown in FIG. 3, the setting assistance device 100 can have an input unit such as a keyboard or a mouse for receiving input such as operations by a user or the like. In addition, the setting assistance device 100 can have a display unit such as a display for displaying setting assistance information to a user or the like.

(Communication unit 110)
The communication unit 110 performs data communication related to the reception of input data and correct answer data, the output of information related to the calculated threshold value, etc. The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and controls communication via an electric communication line such as a LAN (Local Area Network) or the Internet. The communication unit 110 is connected to a network by wire or wirelessly as necessary, and can transmit and receive information in both directions.

(Memory unit 120)
The storage unit 120 stores data and programs used in various processes by the control unit 130, and various data acquired by the operation of the control unit 130. The storage unit 120 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3, the storage unit 120 has an input data DB 121, a correct answer data DB 122, a matching result DB 123, and an inference condition DB 124.

(Input data DB121)
The input data DB 121 is a database that stores input data to be input to a model used in two-layer inference. For example, the input data DB 121 stores still images, moving images (video), and the like as input data to be input to the model.

(Correct answer data DB122)
The supervised data DB 122 is a database that stores supervised data used in a matching process performed by a matching unit 134 described later. For example, the supervised data DB 122 stores, as supervised data, results of object detection (e.g., Bounding Box, etc.) generated by a generating unit 132 described later, results of posture estimation, etc. (e.g., data such as key points and bones), generated data (ground truth) generated in advance by hand or by another information processing device, etc.

(Matching result DB123)
The matching result DB 123 is a database that stores the results of a matching process performed by the matching unit 134 described later. Specifically, the matching result DB 123 stores predetermined indices of accuracy and performance associated with each of the varied thresholds for each combination of models to be used that is included in the specified inference conditions.

Here, an example of the matching result stored by the matching result DB 123 will be described. FIG. 4 is a table diagram showing an example of the matching result according to this embodiment. As shown in FIG. 4, the matching result DB 123 stores the accuracy, performance, recommended threshold value, etc. in association with each varied threshold value.

The thresholds mentioned above include one or more thresholds that are varied based on the inference conditions specified by the user, etc. Precision is an evaluation index related to the result of matching the inference result of the two-layer inference with the correct answer data, and includes the accuracy rate, recall rate, F-measure, etc. Performance is an index that represents the performance of the two-layer inference, such as the proportion of heavy inferences performed using the second-layer model in the two-layer inference, and latency, which is an index of real-time performance. The recommended threshold is a threshold that meets the criteria specified by the parameters included in the inference conditions described later, and is information that identifies whether or not the threshold is recommended to the user, etc.

For example, as shown in FIG. 4, the matching result DB123 stores matching results such as accuracy "80%", performance "65%", and recommended threshold "◯ (recommended)" that correspond to the threshold "0.3" among the varied thresholds. Note that the recommended threshold "×" indicates that it is not a recommended threshold.

(Inference condition DB124)
The inference condition DB 124 is a database that stores inference conditions for creating auxiliary setting information. The inference condition DB 124 stores the inference conditions specified by a user, etc., in association with the model used, input data, correct answer data, evaluation index, and parameters.

Here, an example of an inference condition stored by the inference condition DB 124 will be described. FIG. 5 is a table diagram showing an example of an inference condition according to this embodiment. For example, the inference condition DB 124 stores models to be used, such as a first-layer model used for two-layer inference and a second-layer model used for heavy inference, which are associated with a unique number such as a No. that identifies the inference condition, input data, correct answer data, evaluation indices such as accuracy and performance, and parameters such as target values for the evaluation indices, threshold ranges, and the number of inference iterations.

The above-mentioned model to be used is information indicating which model is to be used among a model used for motion detection, a model used for object detection, a model used for posture estimation, and the like. The input data is information indicating still images, moving images, and the like to be input to a model for performing two-layer inference. The correct answer data is information indicating the correct answer data to be used in the matching process by the matching unit 134 described later. The target value among the parameters is a parameter for determining a standard used when the recommended threshold is generated by the creation unit 135 described later. The threshold range among the parameters is the threshold variation range when the threshold is varied by the inference unit 133 described later to perform light inference. The number of inference repetitions among the parameters is a parameter indicating how many times light inference is performed within the threshold variation range. The accuracy and performance among the evaluation indexes are the same as those described in FIG. 4, and therefore will not be described here.

For example, the inference condition DB124 stores inference conditions such as the first layer model "motion detection model", the second layer model "object detection model", the input data "verification video A", the correct answer data "correct answer data A", the accuracy "correct answer rate", the performance "proportion of frames processed by the first layer model", the target values "accuracy: 70%, performance: 20%", the threshold range "0 to 1", and the number of inference repetitions "10 times", all of which are associated with No. "1".

(Control unit 130)
The control unit 130 has an internal memory for temporarily storing programs and processing data that define various processing procedures of the setting assistance device 100, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in FIG. 3, the control unit 130 has a receiving unit 131, a generating unit 132, an inferring unit 133, a collating unit 134, a creating unit 135, and an output unit 136.

(Reception Unit 131)
The receiving unit 131 receives the designation of inference conditions for creating auxiliary setting information. Specifically, the receiving unit 131 receives the model to be used, the input data, the correct answer data, the evaluation index, and the parameters via the input unit and the communication unit 110 described above, and stores them in the inference condition DB 124.

The reception unit 131 also receives input data used for two-layer inference by the inference unit 133 described below via the communication unit 110 and stores it in the input data DB 121. When the generation unit 132 described below does not generate correct answer data for use in matching processing by the matching unit 134 described below, the reception unit 131 receives separately generated correct answer data via the input unit or communication unit 110 and stores it in the correct answer data DB 122. For example, in the case of the correct answer data "correct answer data C" shown in FIG. 5, the reception unit 131 receives data generated in another information processing device or data generated manually as correct answer data (ground truth) and stores it in the correct answer data DB 122.

(Generation unit 132)
The generating unit 132 generates correct answer data using the second layer model used for heavy inference based on inference conditions specified by a user, etc. For example, in the case of the correct answer data "correct answer data A" shown in Fig. 5, the generating unit 132 generates correct answer data including the same number of positive and negative examples, with frames that detect at least one object being a positive example (Positive) and frames that do not detect an object being a negative example (Negative) among the results of object detection based on the second layer model. Also, in the case of the correct answer data "correct answer data B" shown in Fig. 5, the generating unit 132 generates a Bounding Box as a result of object detection based on the second layer model.

(Inference Unit 133)
The inference unit 133 compares a predetermined value obtained by performing light inference with each of the varied thresholds, and performs heavy inference when a judgment condition for whether to perform heavy inference is satisfied, and outputs an inference result of the two-layer inference. For example, the inference unit 133 performs heavy inference when a difference degree, a confidence degree, or the like obtained as a result of performing inference using the first layer model used for light inference satisfies a judgment condition for each of the varied thresholds, and outputs the obtained result of the two-layer inference.

As an example, the inference unit 133 performs heavy inference when the judgment condition "if the degree of difference calculated by light inference exceeds a threshold (lower threshold)", performs heavy inference, and outputs the obtained result of the two-layer inference to the matching unit 134. Also, the inference unit 133 performs heavy inference when the judgment condition "if the confidence calculated by light inference falls below a threshold (upper threshold)", performs heavy inference, and outputs the obtained result of the two-layer inference to the matching unit 134.

If the obtained difference degree, certainty degree, etc. do not satisfy the judgment condition related to the threshold, the inference unit 133 may output information such as "Judgment condition not satisfied, no output of second-layer inference" as the result of the second-layer inference.

(Collation unit 134)
The collation unit 134 varies the threshold used in the second-layer inference, and performs the second-layer inference for each varied threshold, and collates the result of the second-layer inference for each varied threshold with the correct answer data. Specifically, the collation unit 134 compares the result of the second-layer inference with the correct answer data created in advance, and calculates a predetermined index such as the accuracy of the second-layer inference as a collation result. In addition, the collation unit 134 uses the result of the second-layer inference to calculate a predetermined index such as the rate at which heavy inference is performed using the second-layer model and performance such as latency, which is an index of real-time performance, as a collation result. Then, the collation unit 134 stores the collation result in the collation result DB 123.

For example, when the light inference in the second-layer inference is motion detection and the heavy inference is object detection, the matching unit 134 calculates the accuracy rate as an index of accuracy and the ratio of frames processed in the first-layer model as an index of performance based on the condition No. "1" of the inference conditions shown in FIG. 5. The matching unit 134 then stores the calculated results as matching results in the matching result DB 123. Here, the accuracy rate is the rate of agreement between the inference result by the second-layer inference and the correct data, in other words, it is an index that indicates how accurately the second-layer inference (motion detection, object detection, etc.) is performed on the input data. The ratio of frames processed in the first-layer model is the number of data (frames) processed by only light inference relative to the number of input data (frames).

The accuracy rate mentioned above is calculated, for example, by the formula "Accuracy rate = (TP + TN) ÷ (TP + TN + FP + FN)." TP is the number of frames in which the inference result of light inference in positive examples is "moving object present." FN is the number of frames in which the inference result of light inference in positive examples is "no moving object." FP is the number of frames in which the inference result of light inference in negative examples is "moving object present." TN is the number of frames in which the inference result of light inference in positive examples is "no moving object." Positive examples here are "frames in which at least one object is detected." Negative examples are "frames in which no object is detected."

Specifically, the matching unit 134 compares the result of the two-layer inference of "moving object present (heavy inference performed)" with the correct answer data, and calculates the number of frames that fall into "TP (object present)" and "FP (no object)" among the "frames determined to have a moving object". The matching unit 134 also compares the result of the two-layer inference of "no moving object (no heavy inference performed)" with the correct answer data, and calculates the number of frames that fall into "FP (moving object present)" and "FN (no moving object)" among the "frames determined to have no moving object". The matching unit 134 then applies the number of frames that fall into "TP, TN, FP, FN" to the formula for calculating the accuracy rate described above to calculate the accuracy rate.

Furthermore, the matching unit 134 can calculate the recall rate of positive examples and the recall rate of negative examples. Note that the recall rate of positive examples here is an index showing the degree of missed detections; for example, a higher recall rate of positive examples indicates fewer missed detections. The recall rate of negative examples is an index showing the accuracy rate of frames in which it is determined that an object is not detected; a higher value indicates a more accurate determination (i.e., fewer missed detections).

The recall rate of the positive examples described above is calculated, for example, by the formula "Recall rate of positive examples = TP ÷ (TP + FN)." The recall rate of the negative examples is calculated, for example, by the formula "Recall rate of negative examples = TN ÷ (TN + FP)."

On the other hand, when the light inference is object detection and the heavy inference is object detection, the matching unit 134 calculates the recall as an index of accuracy and the ratio of frames for which processing has ended in the first layer model as an index of performance based on the condition No. "2" of the inference conditions shown in FIG. 5. Then, the matching unit 134 stores the calculated result as the matching result in the matching result DB 123.

The above-mentioned recall rate is calculated based on IoU (Intersection over Union). Specifically, when the result of object detection using heavy inference is regarded as correct answer data, the matching unit 134 calculates the IoU of the bounding box (hereinafter sometimes referred to as "BB") obtained using light inference and the BB obtained using heavy inference. For example, the matching unit 134 calculates the IoU using the formula "IoU = (intersection area of the BB obtained using light inference and the BB obtained using heavy inference) ÷ (total area of the BB obtained using light inference and the BB obtained using heavy inference combined)."

The matching unit 134 determines that detection is successful when the IoU is equal to or greater than the threshold for matching processing, and that detection is unsuccessful when the IoU is less than the threshold for matching processing. Using the above-mentioned determination results of "detection successful" and "unsuccessful detection", the matching unit 134 calculates the recall rate based on the formula "recall rate = (number of successful detections) ÷ (number of successful detections + number of undetected)". Note that the recall rate here indicates the proportion of frames detected (objects detected) by heavy inference out of the number of frames detected (objects detected) by light inference. Also, IoU is an index that indicates the proportion of image overlap, and the larger the IoU, the more images overlap.

The matching unit 134 can also calculate the precision rate and the like. For example, the matching unit 134 can calculate the precision rate using the formula "Precision rate = (number of successful detections) / (number of BBs obtained by light inference)." Note that the precision rate here is the number of BBs that were also detected by heavy inference (number of successful detections) out of the number of BBs obtained by light inference, and is an index showing the degree to which the inference results of light inference and heavy inference match.

(Creation Unit 135)
The creation unit 135 creates a correlation graph between the varied thresholds and the accuracy and performance for each varied threshold calculated as the matching result by the matching unit 134. The creation unit 135 can also create a correlation graph in which at least one of the accuracy and performance calculated as a predetermined index is associated with each varied threshold. The creation unit 135 can also create a correlation graph using the matching results stored in the matching result DB 123.

The creation unit 135 calculates a threshold value corresponding to a specified index as a recommended threshold value when the specified index falls within a range of preset criteria. For example, the creation unit 135 identifies accuracy and performance that meet criteria determined by target values of accuracy and performance specified by a user, etc. Then, the creation unit 135 calculates a threshold value corresponding to the identified accuracy and performance as a recommended threshold value.

In addition, when the average of the specified indexes to which a predetermined weighting is applied falls within a range of a preset standard, the creation unit 135 calculates a threshold value corresponding to the specified index as a recommended threshold value. For example, the creation unit 135 performs weighting such as multiplying the accuracy and performance by a preset coefficient, and calculates a harmonic mean for each of the weighted accuracy and performance. Then, when the calculated harmonic mean satisfies a condition such as exceeding a preset standard, the creation unit 135 calculates a threshold value corresponding to the accuracy and performance related to the harmonic mean as a recommended threshold value.

The creation unit 135 identifies the results of the second-layer inference in which a predetermined change has occurred due to varying the threshold. Specifically, when a change has occurred in the inference result of the second-layer inference (light inference or heavy inference) due to varying the threshold, the creation unit 135 calculates the rate of change between the inference results before and after the change, and identifies the inference result that exceeds a predetermined standard as an inference result in which a change has occurred. The creation unit 135 then creates the result of comparing the inference result before the change with the inference result after the change as setting auxiliary information. Note that an example of the setting auxiliary information created by the creation unit 135 will be described in the section on the output unit 136 below.

(Output unit 136)
The output unit 136 outputs auxiliary setting information such as the correlation graph, the recommended threshold, and the comparison result of the inference result change created by the creation unit 135 to the user or the like via the input unit and the communication unit 110 described above. Here, an example of auxiliary setting information output by the output unit 136 will be described. FIG. 6 is a diagram showing an example of auxiliary setting information output according to this embodiment. FIG. 6 shows a correlation graph (FIG. 6(1)), a recommended threshold (FIG. 6(1-1)), and a comparison result of the inference result change (FIG. 6(2)). The correlation graph as shown in FIG. 6(1) is a graph in which accuracy and performance associated with each threshold value that has been changed are plotted. Then, the correlation graph is displayed with the accuracy value and performance value associated with each threshold value that has been changed being connected by a line segment.

The X-axis of the correlation graph represents the varied threshold value, and is set based on the "threshold range" and "number of times the inference is repeated" of the inference conditions specified by the user, etc. For example, when the threshold range is "0 to 1" and the number of times the inference is repeated is "10 times," the interval for varying the threshold value is set as "0.1."

The Y-axis on the left side of the correlation graph represents accuracy values such as the accuracy rate and recall rate calculated based on the matching results of the matching unit 134. On the other hand, the Y-axis on the right side of the correlation graph represents performance values such as the ratio of frames for which processing is completed with the first-layer model, the ratio at which second-layer inference is performed, and latency, all calculated based on the matching results of the matching unit 134.

Furthermore, the output unit 136 can display a recommended threshold on the correlation graph and output the correlation graph. For example, when the creation unit 135 calculates a recommended threshold based on the inference conditions, the output unit 136 can output a correlation graph with a display indicating that the recommended threshold is "0.3", as shown in (1-1) of FIG. 6.

The comparison result of the inference result change as shown in FIG. 6 (2) is information that is presented to the user, etc., that a change has occurred in the inference result of the two-layer inference due to the change in the threshold. For example, in the upper diagram of the comparison result of the inference result change as shown in FIG. 6 (2), three objects (people) are detected (from (2-1) to (2-3) in FIG. 6). On the other hand, in the lower diagram of the comparison result of the inference result change as shown in FIG. 6 (2), two objects (people) are detected (from (2-4) to (2-5) in FIG. 6). In this way, the output unit 136 can output the comparison result between the upper diagram and the lower diagram, in which the detection results of objects (people) differ due to the change in the threshold.

(First example: When determining whether or not to perform heavy inference based on a lower threshold)
From here, an example of the setting assistance process realized by the above-mentioned functions of the setting assistance device 100 will be described with reference to Fig. 7 to Fig. 13. Fig. 7 to Fig. 13 are diagrams showing an example of setting assistance information generation according to this embodiment.

First, an example of "determining whether or not to perform heavy inference in two-layer inference using a lower threshold" will be described with reference to Figs. 7 to 9. Fig. 7 shows a two-layer inference system 10 that uses a motion detection model as a first-layer model for light inference and an object detection model as a second-layer model for heavy inference, and a setting assistance device 100 that creates setting assistance information for setting thresholds used in the two-layer inference system 10.

In the example shown in FIG. 7, the two-layer inference system 10 performs heavy inference on the cloud side when the degree of difference obtained by light inference on the edge side exceeds a threshold value ("High degree of difference" at the top of FIG. 7). In other words, the two-layer inference system 10 shown in FIG. 7 performs heavy inference only when a certain level of difference occurs for each input data, and detects objects that have been set in advance as detection targets, such as people, animals, and automobiles, from among the input data. In other words, when the tasks processed by the first layer model and the second layer model are different, the two-layer inference system 10 reduces the processing load by reducing the frequency of heavy inference that places a high processing load.

As shown in FIG. 7, the setting assistance device 100 performs two-layer inference using the accepted inference conditions and input data, and creates setting assistance information by comparing the inference results of the two-layer inference with the correct answer data. In the first example, the setting assistance device 100 outputs a correlation graph, a recommended threshold value (recommended threshold value), and the like as setting assistance information.

Next, an example of the setting assistance information output by the setting assistance device 100 in the first example will be described. FIG. 8 shows an example of a correlation graph output by the setting assistance device 100. In the correlation graph shown in FIG. 8, the X-axis represents the varied threshold value, the Y-axis (left) represents the accuracy rate as an index of precision, and the Y-axis (right) represents the ratio of frames at which processing is completed in the first layer model as an index of performance.

As described above, in the first example, the two-layer inference system 10 performs heavy inference when the degree of difference obtained by light inference exceeds a threshold (lower threshold). That is, in the first example, as the value of the threshold increases, the frequency with which the degree of difference obtained by light inference exceeds the threshold decreases, and the frequency with which heavy inference is performed decreases. In this way, since the frequency with which heavy inference is performed by the second-layer model decreases as the threshold increases, the correlation graph shown in Figure 8 shows a tendency for the accuracy rate to decrease and the proportion of frames for which processing is completed by the first-layer model to increase.

The setting assistance device 100 can also display an operation screen 20 that accepts "selection of evaluation index" when outputting the correlation graph. This operation screen 20 includes check boxes for selecting evaluation indexes to be displayed in the correlation graph, such as "accuracy rate," "recall rate," and "proportion of frames that have been processed in the first layer model." In the example of FIG. 8, "accuracy rate" and "proportion of frames that have been processed in the first layer model" have been selected.

The setting assistance device 100 also outputs a correlation graph displaying recommended thresholds to be presented to the user, etc., as shown in FIG. 9. Specifically, the setting assistance device 100 can calculate, as the recommended threshold, a threshold that satisfies the criteria determined by the target value of the accuracy and the target value of the performance set by the user, etc.

For example, when the target value of the accuracy rate is set to "70% (or more)" as shown in the operation screen 21 of FIG. 9, the setting assistance device 100 identifies the threshold value "0 to 0.5" corresponding to the section indicated by the solid line where the accuracy rate corresponding to the Y axis (left) is 70% or more as a recommended threshold candidate. When the target value of the ratio of processing end frames in the first layer model is set to "20% (or more)", the setting assistance device 100 identifies the threshold value "0.3 to 1" corresponding to the section indicated by the solid line where the ratio of processing end frames in the first layer model corresponding to the Y axis (right) is 20% or more as a recommended threshold candidate. Then, the setting assistance device 100 calculates the threshold value "0.3 to 0.5" that satisfies both the target value of the accuracy rate and the target value of the ratio of processing end frames in the first layer model as a recommended threshold, as shown in FIG. 9.

In addition, the setting assistance device 100 can calculate the recommended threshold value to be "0.3", which provides higher accuracy when "preferential accuracy" is selected, and "0.5", which provides higher performance when "preferential performance" is selected.

The setting assistance device 100 can also output a correlation graph that displays the breakdown values for each index. Specifically, the setting assistance device 100 can display the "reproduction rate of positive examples" and the "reproduction rate of negative examples" as breakdown values at each point of any accuracy rate. For example, as shown in FIG. 9, the setting assistance device 100 can output a correlation graph that displays breakdown values such as a positive example reproduction rate of "80%" and a negative example reproduction rate of "40%" as indicators of an accuracy rate of "80%" at a threshold value of 0.3.

(Second example: When determining whether or not to perform heavy inference in two-layer inference using an upper threshold)
Next, an example of "determining whether or not to perform heavy inference in two-layer inference using an upper limit threshold" will be described with reference to Figures 10 and 11. Figure 10 shows a two-layer inference system 10 that uses an object detection model as a first-layer model for light inference and an object detection model as a second-layer model for heavy inference, and a setting assistant device 100 that creates setting assistant information for setting thresholds used in the two-layer inference system 10.

In the example shown in FIG. 10, the two-layer inference system 10 does not perform heavy inference on the cloud side when the confidence obtained by the light inference on the edge side exceeds a threshold, and performs heavy inference on the cloud side when the confidence is below the threshold. In other words, the two-layer inference system 10 shown in FIG. 10 uses the inference result of the light inference as it is as the final output result of the two-layer inference when the confidence of the inference result of the light inference exceeds a threshold ("high confidence" at the top of FIG. 10). On the other hand, when the confidence is below the threshold ("low confidence" at the top of FIG. 10), it determines that the inference result of the light inference is insufficient, performs heavy inference, and outputs the result of the heavy inference as the result of the two-layer inference. In other words, when the tasks processed by the first layer model and the second layer model are the same, the two-layer inference system 10 reduces the frequency of performing heavy inference, which has a high processing load, thereby reducing the processing load.

As shown in FIG. 10, the setting assistance device 100 performs two-layer inference using the accepted inference conditions and input data, and compares the results of the two-layer inference with the correct answer data to create setting assistance information. In the second example, the setting assistance device 100 outputs, as setting assistance information, a correlation graph that associates the accuracy and performance of the two-layer inference for each threshold, a recommended threshold (recommended threshold), comparative information on changes in inference results, and the like.

Next, an example of the setting assistance information output by the setting assistance device 100 in the second example will be described. FIG. 11 shows an example of a correlation graph output by the setting assistance device 100. In the correlation graph shown in FIG. 11, the X-axis represents the varied threshold value, the Y-axis (left) represents the recall rate as an index of precision, and the Y-axis (right) represents the ratio of frames processed in the first layer model as an index of performance.

As described above, in the second example, the two-layer inference system 10 performs heavy inference when the confidence obtained by light inference falls below a threshold (upper threshold). That is, in the second example, as the threshold value increases, the frequency with which the confidence obtained by light inference falls below the threshold increases, and the frequency with which heavy inference is performed increases. In this way, as the threshold value increases, the frequency with which heavy inference is performed by the second-layer model increases, and the correlation graph shown in FIG. 11 shows a tendency for the recall rate to gradually increase and the proportion of frames for which processing is completed by the first-layer model to gradually decrease.

The setting assistance device 100 can also display an operation screen 22 that accepts "selection of evaluation index" when outputting the correlation graph. This operation screen 22 includes check boxes for selecting evaluation indexes to be displayed in the correlation graph, such as "correct rate," "recall rate," and "proportion of frames that have been processed in the first layer model." In the example of FIG. 11, "recall rate" and "proportion of frames that have been processed in the first layer model" have been selected.

The setting assistance device 100 also outputs a correlation graph showing recommended thresholds to be presented to the user, etc., as shown in FIG. 12. For example, when the target value of the reproducibility is set to "75% (or more)" as shown in the operation screen 23 of FIG. 12, the setting assistance device 100 specifies the threshold "0.7 to 1" corresponding to the section indicated by the solid line where the reproducibility corresponding to the left of the Y axis is 75% or more, as a recommended threshold candidate. On the other hand, when the target value of the ratio of the processing end frame in the first layer model is set to "30% (or more)" as shown in FIG. 12, the setting assistance device 100 specifies the threshold "0 to 0.7" corresponding to the section indicated by the solid line where the ratio of the processing end frame in the first layer model corresponding to the right of the Y axis is 30% or more, as a recommended threshold candidate. Then, the setting assistance device 100 calculates the threshold "0.7" that satisfies both the target value of the accuracy and the target value of the ratio of the processing end frame in the first layer model as a recommended accuracy-oriented threshold, as shown in FIG. 12.

Note that the "comparison information on changes in inference results" in the second example described above is the same as that described in FIG. 6, so a description of it will be omitted here.

(Third example: When determining the necessity of performing heavy inference that processes light inference results in two-layer inference based on a lower threshold)
Next, an example of "a case in which the necessity of performing heavy inference for processing light inference results in two-layer inference is determined using a lower threshold" will be described with reference to Fig. 13. Fig. 13 shows a two-layer inference system 10 that uses an object detection model as a first-layer model for light inference and a posture estimation model as a second-layer model for heavy inference, and a setting assistance device 100 that creates setting assistance information used to set thresholds used in the two-layer inference system 10.

In the example shown in FIG. 13, the two-layer inference system 10 performs heavy inference on the cloud side when the confidence level obtained by light inference on the edge side exceeds a threshold value ("High confidence level" at the top of FIG. 13). In other words, the two-layer inference system 10 shown in FIG. 13 performs heavy inference when the confidence level for an object extracted from input data exceeds a threshold value, and estimates the bones, etc., of the object detected by light inference. In other words, when the tasks processed by the first layer model and the second layer model are different, the two-layer inference system 10 reduces the processing load by reducing the frequency of heavy inference, which places a high processing load.

Also, as shown in FIG. 13, the setting assistance device 100 performs two-layer inference using the received inference conditions and input data, and creates setting assistance information by comparing the two-layer inference result with the correct answer data. In the third example, the setting assistance device 100 outputs a correlation graph and a recommended threshold (recommended threshold) as setting assistance information. Note that in the third example, since the result of light inference is used to perform heavy inference, the inference result of heavy inference cannot be used as correct answer data, so generated data generated separately is used as correct answer data.

Note that the configuration auxiliary information generated in the third example is similar to that in the first or second example, so a detailed description will be omitted.

(Procedure for setting assistance process)
From here, a description will be given of the procedure of the setting assistance process realized by the setting assistance device 100 according to this embodiment. Fig. 14 is a diagram showing an example of a flowchart of the setting assistance process according to this embodiment.

The reception unit 131 receives the specification of inference conditions (S101).

Here, if correct answer data is not to be generated (No in S102), the receiving unit 131 receives pre-created correct answer data (S103). On the other hand, if correct answer data is to be generated (Yes in S102), the generating unit 132 generates correct answer data using the output result of the heavy inference (S104).

The inference unit 133 performs two-layer inference while varying the threshold based on the inference conditions (S105). Next, the comparison unit 134 compares the inference result of the two-layer inference with the correct answer data (S106). Then, the creation unit 135 creates a correlation graph (S107).

Here, if a recommended threshold is to be created (Yes in S108), the creation unit 135 creates a recommended threshold (S109). On the other hand, if a recommended threshold is not to be created (No in S108), the creation unit 135 skips the step of S109.

In addition, when generating a comparison result regarding the variation in the inference result of the two-layer inference due to the variation in the threshold value (Yes in S110), the creation unit 135 creates the comparison result (S111). On the other hand, when not generating a comparison result (No in S110), the creation unit 135 skips the step of S111.

The output unit 136 outputs the results generated by the creation unit 135 (S112). Then, the setting assistance device 100 ends the process.

(effect)
From here, the effect of the setting assistance device 100 according to this embodiment will be described. The collation unit 134 according to this embodiment varies the threshold value used in the two-layer inference, and performs two-layer inference for each varied threshold value, and collates the results of the two-layer inference for each varied threshold value with the correct answer data. Then, the creation unit 135 creates a correlation graph between the varied threshold value and the accuracy and performance of the two-layer inference for each varied threshold value calculated as the collation result. Therefore, the setting assistance device 100 according to this embodiment has the effect of facilitating tuning of the threshold value used in the two-layer inference.

Specifically, the inference unit 133 compares a predetermined value obtained by performing light inference with each of the varied thresholds, and performs heavy inference when the judgment condition for performing heavy inference is met, and outputs the obtained result of the second-layer inference. The collation unit 134 compares the result of the second-layer inference with the correct answer data created in advance, and calculates a predetermined index including the accuracy of the second-layer inference and the performance of the second-layer inference as the collation result. The creation unit 135 creates a correlation graph in which the accuracy and performance calculated as the predetermined index are associated with each of the varied thresholds. In this way, the setting assistance device 100 automatically varies the thresholds based on the inference conditions specified by the user, etc., and compares the result of the second-layer inference at each of the varied thresholds with the correct answer data to calculate indexes such as accuracy and performance, and creates a correlation graph. Therefore, the setting assistance device 100 has the effect of making it easier to tune the thresholds used in the second-layer inference without the need for the user, etc. to individually check the balance between accuracy and performance when tuning the thresholds.

The creation unit 135 also calculates a threshold value corresponding to a specific index as a recommended threshold value when the specific index falls within a range of preset criteria. The creation unit 135 also calculates a threshold value corresponding to a specific index as a recommended threshold value when an average of a specific weighted specific index falls within a range of preset criteria. In this way, the setting assistance device 100 calculates a candidate threshold value desired by the user as a recommended threshold value. Therefore, the setting assistance device 100 has the effect of reducing the labor required for tuning threshold values performed by a user or the like, and facilitating tuning of threshold values used in two-layer inference.

The creation unit 135 also identifies the results of two-layer inference where a specific change has occurred due to varying the threshold. In this way, the setting assistance device 100 identifies the changed inference result when a change has occurred in the inference result as a result of varying the threshold, thereby enabling the user to visually grasp what change has occurred in the inference result due to the variation in the threshold. As a result, the setting assistance device 100 has the effect of making it easy for the user to select an appropriate threshold.

As described above, the setting assistance device 100 according to this embodiment allows users to easily tune the thresholds to suit the inference use case. As a result, the setting assistance device 100 improves the availability and usability of the two-tier inference system, and enables easy use of the two-tier inference system, regardless of the user's knowledge or level of familiarity with inference.

Furthermore, whereas previously users tuned thresholds by checking the balance between accuracy and performance one by one, the setting assistance device 100 automatically creates a correlation graph that associates thresholds, accuracy, and performance, eliminating the need for individual verification for tuning thresholds and reducing the labor and computer processing required for the tuning.

<Modification>
The following describes modified examples realized by the setting assistance device 100 according to this embodiment.

(Data, etc.)
The input data, correct data, matching results, inference conditions, light inference, heavy inference, performance, accuracy, names of functional parts of the setting assistance device 100, steps, processes, names of steps or processes, etc. used in the description of the above embodiments are merely examples and can be changed as desired.

In addition to still images and video images, the input data DB 121 can also store any data that can be used for model-based inference, such as audio data and text data.

Although the correct answer data DB 122 has been described as storing correct answer data including positive examples and negative examples, this is not limited to this, and any data that can be used for matching processing by the matching unit 134 can be stored without limitation.

Although it has been described that the matching result DB123 stores a correspondence between thresholds, accuracy, performance, recommended thresholds, etc., it may also store matching results that do not include recommended thresholds, for example.

Although the inference condition DB124 has been described as storing the motion detection model, object detection model, and specified estimation model as the models used, it is not limited to these and can store other inference models. Furthermore, the contents of the correct answer data, accuracy, performance, target value, threshold range, number of inference repetitions, etc. stored by the inference condition DB124 may be changed as appropriate according to the actual usage situation. Furthermore, the "target value" among the inference conditions stored by the inference condition DB124 may include any condition such as "greater than or equal to the target value," "less than or equal to the target value," "exceeding the target value," "less than the target value," etc.

As an example, the inference unit 133 has been described as outputting a light inference result when a judgment condition is satisfied, such as "if the degree of difference exceeds a threshold (lower threshold), heavy inference is performed, which is inference using the second layer model" or "if the confidence level is below a threshold (upper threshold), light inference is performed, which is inference using the second layer model." However, the above-mentioned judgment conditions are not limited to these, and it is possible to determine whether or not to output a light inference result based on any condition.

(Flowcharts, etc.)
The steps in the flowcharts and the like may be interchanged as long as there is no contradiction, and some steps may not be performed. In addition, conjunctions such as "next,""continue,""further,""at this time," and "on this occasion" in the explanation of the flowcharts do not limit the order or timing of the execution of the processes in the flowcharts.

(system)
The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

Furthermore, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

<Hardware Configuration>
Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the various processes described in this embodiment, all or part of the processes described as being performed automatically can also be performed manually using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the drawings can be changed as desired unless otherwise specified.

<Program>
In one embodiment, the various devices constituting the setting assistance device 100 can be implemented by installing a setting assistance program as package software or online software on a desired computer. For example, the setting assistance program can be executed by an information processing device to function as the various devices constituting the setting assistance device 100. The information processing device referred to here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones and mobile phones, and slate terminals such as PDAs (Personal Digital Assistants).

FIG. 15 is a diagram showing an example of a computer that executes the setting assistance program according to this embodiment. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the various devices that make up the setting assistance device 100 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to the functional configurations of the various devices that make up the setting assistance device 100 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Furthermore, the setting data used in the processing of the above-mentioned embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the processing of the above-mentioned embodiment.

The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN or a WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

＜Other＞
Although the present embodiment has been described above, the present embodiment is not limited by the description and drawings that form a part of the disclosure. In other words, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are all included in the scope of the present embodiment.

100 Setting assistant device 110 Communication unit 120 Storage unit 121 Input data DB
122 Answer Data DB
123 Matching result DB
124 Inference condition DB
130 Control unit 131 Reception unit 132 Generation unit 133 Inference unit 134 Collation unit 135 Creation unit 136 Output unit

Claims (7)

a collation unit that, when an inference result of a first inference with a lighter processing load satisfies a threshold condition, varies the threshold used in a second layer inference that performs a second inference with a heavier processing load than the first inference, and performs the second layer inference for each varied threshold to compare the result of the second layer inference for each varied threshold with correct answer data;
a generation unit that generates a correlation graph between the threshold value that has been varied and the accuracy and performance of the two-layer inference for each of the threshold values that has been varied and that is calculated as a matching result by the matching unit;
A setting assistance device comprising: an inference unit that compares a predetermined value obtained by performing the first inference with each of the varied thresholds, and performs the second inference and outputs a result of the two-layer inference when a determination condition for whether or not to perform the second inference is satisfied;
The collation unit is
comparing the result of the two-layer inference with the correct answer data created in advance, and calculating a predetermined index including accuracy and performance of the two-layer inference as the matching result;
2. The setting assistance device according to claim 1. The creation unit is
When the predetermined index is within a range of a preset reference, the threshold value corresponding to the predetermined index is calculated as the recommended threshold value.
3. The setting assistance device according to claim 2. The creation unit is
When an average of the predetermined indexes weighted by a predetermined weighting falls within a range of a preset reference, the threshold value corresponding to the predetermined index is calculated as a recommended threshold value.
3. The setting assistance device according to claim 2. The creation unit is
Identifying the results of the two-layer inference in which a predetermined change has occurred due to varying the threshold value;
5. The setting assistance device according to claim 1, wherein the setting assistance device is a setting assistance device for setting a setting value. A setting assistance method to be executed by a setting assistance device,
a comparison process for comparing the result of the second-layer inference for each of the varied thresholds obtained by varying the threshold used in the second-layer inference for performing a second inference for which the processing load is heavier than that of the first inference when the inference result of the first inference for which the processing load is light satisfies a threshold condition and performing the second-layer inference for each of the varied thresholds with the correct answer data;
a generation process for generating a correlation graph between the threshold value that has been varied and the accuracy and performance of the two-layer inference for each of the threshold values that has been varied and that is calculated as a matching result by the matching process;
A setting assistance method comprising: a comparison step of comparing a result of the second-layer inference for each of the varied thresholds obtained by varying the threshold used in the second-layer inference for performing a second inference for which the processing load is heavier than that of the first inference when the inference result of the first inference for which the processing load is light satisfies a threshold condition and performing the second-layer inference for each of the varied thresholds with the correct answer data;
a creating step of creating a correlation graph between the threshold value that has been varied and the accuracy and performance of the two-layer inference for each of the threshold values that has been varied and that is calculated as a matching result by the matching step;
A setting assistance program for causing a computer to execute the above steps.

Images