WO2024236748A1 - Abnormality determination device and abnormality determination method - Google Patents

Abstract

An abnormality determination device (1000) comprises: an image acquisition unit (1010) that acquires and outputs an image; a neural network unit (1020) that has a plurality of layers in a neural network, the neural network unit acquiring the image outputted by the image acquisition unit and outputting a determination result indicating the state of a determination object included in the image; an output storage unit (1050) that stores the output data outputted by the neural network unit; a state comparison unit (1030) that, while a reference value is preliminarily stored, the reference value being a reference for an input value to a first layer that is at least one layer among the plurality of layers of the neural network unit, determines a change in the state of the determination object by using a difference value between the reference value and a current value that is to be newly inputted to the first layer; and a processing control unit (1040) that instructs the neural network unit to not execute processing in layers after the first layer, and instructs that the output data stored in the output storage unit is be outputted, in cases where it is determined by the state comparison unit that the state of the determination object has not changed.

Description

Abnormality determination device and abnormality determination method

The disclosed technology relates to an abnormality determination technology that determines the condition of a subject.

Among the abnormality determination techniques, there are those that determine the condition of a person to be determined by using the results of measurements taken by a sensor device (such as a camera, radar, or infrared sensor).
Among such abnormality judgment technologies, for example, in abnormality judgment technologies that judge the state of a subject of judgment, such as a driver (occupant) of a vehicle (mobile body), since this concerns the safety of the subject of judgment, it is desirable to make the judgment with high accuracy to avoid erroneous judgments.
In response to this, in recent years, research into anomaly determination technology using deep learning has been progressing. Deep learning uses a mathematical model called a neural network (NN). A neural network outputs results through layers such as an input layer, a hidden layer (intermediate layer), and an output layer.
Patent Document 1 discloses an identification device that uses a convolutional neural network (CNN), which is a type of neural network, to identify when a driver of a mobile object is in an abnormal state.
Specifically, in the process of identifying the driver's condition (e.g., whether or not the driver is in an abnormal state) based on an acquired image, the identification device of Patent Document 1 obtains the driver's skeletal information from the image using a convolutional neural network.
The identification device of Patent Document 1 improves the accuracy of the driver's skeletal information by using a neural network.

JP 2020-052869 A

On the other hand, in the abnormality determination technology for a moving object as described above, there is often a demand for rapid determination as well as high accuracy.
When attempting to achieve high accuracy using the above-mentioned neural network in anomaly judgment technology, for example, a method can be adopted in which the neural network extracts and uses a large number of judgment indices indicating the state of the person being judged from an image (for example, a large number of types of indices such as the open/closed state of the eyelids, the number of blinks, the position of the face, the state of brain waves, the heat dissipation state, driving performance (reaction time, etc.)).
If one were to use the above method to output highly accurate abnormality determination results using a neural network, the number of layers in the neural network would increase or the configuration would become complex, resulting in an increase in the amount and time of calculations.
Therefore, there has been a problem with anomaly determination technology in that it is difficult to quickly output highly accurate determination results using a neural network.
The classification device of Patent Document 1 merely outputs skeletal information from an image via a neural network, and is not capable of solving the above-mentioned problems.

The present disclosure aims to solve the above problems and to enable anomaly detection technology to quickly output highly accurate detection results using a neural network.

The abnormality determination device of the present disclosure is
an image acquisition unit that acquires and outputs an image;
a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image;
an output storage unit that stores output data output by the neural network unit;
a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
a processing control unit that instructs the neural network unit not to execute processing in the first layer and subsequent layers when the state comparison unit determines that the state of the object to be determined has not changed, and instructs the neural network unit to output the output data stored in the output storage unit;
Equipped with:

The present disclosure has the effect of enabling anomaly detection technology to quickly output highly accurate detection results using a neural network.

FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to a first embodiment of the present disclosure. FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure. FIG. 3 is a flowchart showing an example of more detailed processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure. FIG. 4 is a diagram showing an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to the second embodiment of the present disclosure. FIG. 5 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the feature extraction unit 1300B in the abnormality determination device 1000B. FIG. 6 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the abnormality state determination unit 1500B in the abnormality determination device 1000B. FIG. 7 is a flowchart showing an example of processing by the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. FIG. 8 is a flowchart showing an example of more detailed processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure. FIG. 9 is a flowchart showing an example of a process of the feature extraction unit 1300B in a case where the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341. FIG. 10 is a flowchart illustrating an example of a process of the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit 1030 (first state comparison unit 1030B) is a convolutional layer state comparison unit. FIG. 11 is a flowchart illustrating an example of a process performed by the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit 1030 (first state comparison unit 1030B) is a pooling layer state comparison unit. FIG. 12 is a flowchart showing an example of processing of the abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is the first fully connected layer state comparison unit 1541. FIG. 13 is a flowchart showing an example of processing by the abnormal state determination unit 1500B in the second embodiment of the present disclosure when the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B is a second fully connected layer state comparison unit. FIG. 14 is a diagram showing an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C. FIG. 16 is a diagram showing an example of the internal configuration of the abnormality warning device 100C and the abnormality state determination unit 1500C in the abnormality determination device 1000C. FIG. 17 is a flowchart showing an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 18 is a flowchart showing a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 19 is a flowchart showing a detailed first example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 21 is a flowchart showing a second detailed example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure. FIG. 22 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the present disclosure. FIG. 23 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the present disclosure.

In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.
In the following description, ordinal numbers such as "first,""second," and "third" may be used. These terms are used for convenience to facilitate understanding of the contents of the embodiments, and are not limited to the order that may result from these ordinal numbers.

Embodiment 1.
In the first embodiment, one mode of a basic configuration of the present disclosure will be described.

An example of the configuration of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment will be described.
FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to a first embodiment of the present disclosure.
The abnormality warning device 100A acquires an image, uses the image to judge the state of an object to be judged that is captured in the image, and outputs a warning if the state of the object to be judged indicates an abnormality.
The abnormality warning device 100 shown in FIG. 1 includes an abnormality determination device 1000A and a warning output unit 2000.

The abnormality determination device 1000A obtains an image and uses the image to output the state of the object to be determined captured in the image. The state of the object to be determined is output data including a value in the form of, for example, a determination value or a probability value.
The abnormality determination device 1000A includes an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, and an output storage unit 1050.

The image acquisition unit 1010 acquires and outputs an image.
The image acquisition unit 1010 acquires an image by inputting an image (moving image or still image) captured by a camera, for example.
The camera is, for example, an imaging device installed inside the vehicle cabin, which captures an image of a determination target present inside the vehicle cabin (the determination target is, for example, a living body such as a passenger including the driver) and outputs the captured image.
When the image is a moving image, the moving image is divided by a camera into still images (frames) at regular time intervals, and the divided images are input to the image acquisition unit 1010 .

The neural network unit 1020 is composed of a neural network.
The neural network unit 1020 has multiple layers in a neural network, acquires the image output by the image acquisition unit 1010, and outputs output data that is a judgment result indicating the state of the judgment target contained in the image.
The multiple layers in a neural network are an input layer, a hidden layer (an intermediate layer), and an output layer, and more specifically, layers such as a convolutional layer and a pooling layer.
Here, a neural network with many layers (deep) is particularly called a DNN (Deep Neural Network), and its derivatives include, for example, a CNN (Convolutional Neural Network) and an RNN (Recurrent Neural Network). CNN is often used in the fields of object recognition and image recognition, while RNN is often used in time series processing, voice recognition, natural language processing, and the like. The neural network in the present disclosure is applicable regardless of the form of the neural network. In this description, the form of CNN will be described as an example.

The state comparison unit 1030 judges a change in the state of the object to be judged.
The state comparison unit 1030 pre-stores a reference value that is a standard for the input value to the first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be judged using the difference value between the reference value and the current value, which is the value that is to be newly input to the first layer.
The first layer may be a predetermined one of all layers in the neural network unit 1020, a predetermined portion of all layers, or all layers.

The reference value is a previous value, which is an input value of at least one layer, the first layer, stored as a result of previous processing by multiple layers of the neural network unit 1020, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer. In the explanation of the processing described later, a case in which the previous value is used as the reference value will be explained.
However, the following values may be stored and used as the reference values.

The reference value can use typical processing results that have been learned and modeled in advance.
In other words, the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
By using typical processing results, comparisons can be made without being influenced by noise or anomalies contained in the actual observed images.

In addition, the reference value may use a part of the results determined based on advance information.
The partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head.
Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network.
This allows for reduced memory usage and data handling, resulting in faster processing speeds.

The state comparison unit 1030 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison unit 1030 determines that the state of the determination target is unchanged.
More specifically, the state comparison unit 1030 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.

In response to changes in the state of the object to be judged, the processing control unit 1040 issues a command to restrict the current processing operation in the neural network unit 1020, and also issues a command to output the output data output by the previous processing of the neural network unit 1020.
When the state comparison unit 1030 determines that the state of the object to be judged has not changed (no change includes cases where there is little change), the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in at least one layer out of the multiple layers, that is, the first layer or subsequent layers, and also instructs the neural network unit 1020 to output the output data stored in the output storage unit 1050.
The output data is, for example, a judgment value that is a value indicating the state of the person to be judged. In the embodiment of the present disclosure, the judgment value is also referred to as a probability value that includes a probabilistic element.

The output storage section 1050 stores the output data (decision value (probability value)) output by the neural network section 1020 .
When the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data.
The output storage unit 1050 outputs the output data to the warning output unit 2000, for example.
The output storage unit 1050 holds the latest output data output as a result of processing performed by the neural network unit 1020. For example, every time the output storage unit 1050 obtains output data output by the neural network unit 1020, it erases the output data previously stored and stores the latest output data.

The alarm output unit 2000 acquires output data, and outputs an alarm signal based on the output data to an alarm device (not shown) or the like.
The warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.

The abnormality determination device 1000A shown in FIG. 1 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000A is equivalent to the abnormality warning device 100A shown in FIG. 14. In the following explanation, the abnormality determination device 1000A will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100A and the abnormality determination device 1000A.

In addition to the above configuration, abnormality determination device 1000A may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
A control unit (not shown) controls the entire abnormality determination device 1000A and each of its components. The control unit (not shown) starts up the abnormality determination device 1000A in response to, for example, an external command. The control unit (not shown) also controls the state (operation state = start-up, shutdown, sleep, etc.) of the abnormality determination device 1000A.
A storage unit (not shown) stores each piece of data used in the abnormality determination device 1000A. The storage unit (not shown) stores, for example, outputs (output data) from each component in the abnormality determination device 1000A, and outputs data requested by each component to the component that has made the request.
A communication unit (not shown) communicates with an external device. For example, communication is performed between the abnormality determination device 1000A and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000A does not have a display unit or an audio output unit, communication is performed between the abnormality determination device 1000A and an external device such as a display unit or an audio output device.

An example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment will be described.
FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
The abnormality determination device 1000A starts the process shown in FIG. 2 when an image is input from a camera, for example.

Abnormality determination device 1000A executes image acquisition processing (step ST100).
In the image acquisition process, the image acquisition unit 1010 of the abnormality determination device 1000A acquires and outputs an image.

Abnormality determination device 1000A executes a state storage and state comparison process (step ST200).
In the state storage and state comparison process, the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of at least one layer, that is, the first layer, among the processing results in all layers of the neural network unit 1020 for at least the first time after the start of processing.
The state comparison unit 1030 does not perform state comparison processing for the first processing, but simply stores the previous value, but judges whether there has been a change in the state of the object to be judged from the second processing onwards.
The state comparison unit 1030 pre-stores a reference value that is a standard for the input value to the first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be judged using the difference value between the reference value and the current value, which is the value that is to be newly input to the first layer.

Abnormality determination device 1000A executes a process control process (step ST300).
In the process control process, the process control unit 1040 of the abnormality determination device 1000A issues a command to restrict the current processing operation in the neural network unit 1020 in response to a change in the state of the object to be determined, and also issues a command to output the output data output by the previous processing of the neural network unit 1020.
Specifically, when the state comparison unit 1030 determines that the state of the object to be judged has not changed (no change includes cases where there is little change), the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in at least one layer out of the multiple layers, that is, the first layer or subsequent layers, and also instructs the processing control unit 1040 to output the output data stored in the output storage unit 1050.

Abnormality determination device 1000A executes a state output process (step ST400).
In the state output process, the neural network unit 1020 or the output storage unit 1050 of the abnormality determination device 1000A outputs output data.
When processing has been performed in all layers, the neural network unit 1020 outputs output data to the warning output unit 2000.
When the output storage unit 1050 receives a command from the process control unit 1040 , it outputs the latest stored output data to the alarm output unit 2000 .

Abnormality determination device 1000A executes an alarm output process (step ST500).
In the alarm output process, the alarm output unit 2000 of the abnormality determination device 1000A acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc. The alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.

When the abnormality determination device 1000A executes the process of step ST500,
The series of processes shown in FIG. 2 ends, and the process is repeated from step ST100.
For example, when the camera is turned off, the abnormality determination device 1000A is also turned off in conjunction with the camera.

A detailed example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment will be described.
FIG. 3 is a flowchart showing an example of more detailed processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
When abnormality determination device 1000A starts the process shown in FIG. 3, first, similarly to step ST100 described above, it executes image acquisition processing (step ST100).

Abnormality determination device 1000A executes a storage process (step ST201).
In the storage process, the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of at least one layer, the first layer, among the processing results in all layers of the neural network unit 1020 for at least the first time after the start of processing.
Furthermore, the state comparison unit 1030 stores the input values of the first layer each time processing is performed in the first layer.

The state comparison unit 1030 of the abnormality determination device 1000A executes a previous storage determination process to determine whether the data has been stored previously (step ST202).

The state comparison unit 1030 of the abnormality determination device 1000A executes a comparison process between the current value and the previous value (step ST203). The state comparison unit 1030 calculates the difference (difference value) between the current value, which is the value that is about to be newly input to the first layer, and the previous value (reference value).

State comparison section 1030 of abnormality determination device 1000A determines a change in the state of the object to be determined using the difference value (step ST204).
The state comparison unit 1030 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison unit 1030 determines that the state of the determination target is unchanged.
More specifically, the state comparison unit 1030 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.

If the state comparison unit 1030 of the abnormality determination device 1000A determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed (step ST204 "NO"), the processing control unit 1040 of the abnormality determination device 1000A executes a normal processing command process (step ST301). The processing control unit 1040 outputs a command to execute processing in the first layer of the neural network unit 1020.

The neural network unit 1020 of the abnormality determination device 1000A executes output processing (step ST401). The neural network unit 1020 outputs the output data to the alarm output unit 2000.

Output storage unit 1050 of abnormality determination device 1000A executes an output storage process (step ST402).
The output storage section 1050 stores the output data output by the neural network section 1020 .

If the state comparison unit 1030 of the abnormality determination device 1000A determines that the difference is less than the threshold value and that the state to be determined has not changed (step ST204 ``YES''), the processing control unit 1040 of the abnormality determination device 1000A executes an omission processing command processing (step ST302).
In the omission processing command processing, the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in the first layer or subsequent layers, which is at least one layer among the multiple layers, and also instructs the output storage unit 1050 to output the output data stored therein.

The output storage unit 1050 of the abnormality determination device 1000A executes a stored data output process (step ST403). When the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data to the alarm output unit 2000.

The alarm output unit 2000 executes an alarm output process (step ST500).
The alarm output unit 2000 determines whether to output an alarm based on the judgment value contained in the output data, similar to the processing of step ST500 already described, and if it determines to output an alarm, outputs an alarm signal to an alarm device or the like (not shown).

When abnormality determination device 1000A executes the process of step ST500, it ends the series of processes shown in FIG. 3 and repeats the process from step ST100.
For example, when the camera is turned off, the abnormality determination device 1000A is also turned off in conjunction with the camera.

The configuration and processing described above allows the anomaly determination device to omit processing of layers in the neural network depending on the conditions.

The abnormality determination device of the present disclosure has the following configuration.

"An image acquisition unit that acquires and outputs an image;
a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image;
an output storage unit that stores output data (judgment value (probability value)) output by the neural network unit;
a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
a processing control unit that instructs the neural network unit not to execute processing in the first layer and subsequent layers when the state comparison unit determines that the state of the object to be determined has not changed, and instructs the neural network unit to output the output data stored in the output storage unit;
An abnormality determination device equipped with the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.

The abnormality determination method of the present disclosure has the following configuration.

The image acquisition unit outputs the acquired image to the neural network unit,
the neural network unit having a plurality of layers in a neural network acquires the image output by the image acquisition unit, and outputs a judgment result indicating a state of a judgment target included in the image;
an output storage unit stores the output data (judgment value (probability value)) output by the neural network unit;
a state comparison unit pre-stores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
When the state comparison unit determines that the state of the object to be determined has not changed, the processing control unit instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs the neural network unit to output the output data stored in the output storage unit.
Method for determining abnormal conditions.

As a result, the present disclosure has the effect of providing an anomaly determination method that makes it possible to quickly output highly accurate determination results using a neural network.

The abnormality determination device of the present disclosure is further configured as follows.

"The anomaly determination device, wherein the reference value is a previous value stored for each layer as a result of the previous processing in the neural network."

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The reference value is a typical output value of a pre-modeled intermediate layer in the neural network.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The reference value is an output value of a predetermined node among the nodes included in the neural network.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the above abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The state comparison unit is
a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value;
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The state comparison unit is
When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The state comparison unit is
When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The present invention further includes an alarm output unit that acquires the output data, which is a judgment value indicating a drowsy state or a dozing state of a person to be judged, and outputs an alarm to the person to be judged according to the judgment value.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

Embodiment 2.
In the second embodiment, a form in which the basic mechanism of the present disclosure is applied to layers in the neural networks of a feature extraction unit and an abnormal state determination unit will be described.

An example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to the second embodiment will be described.
FIG. 4 is a diagram showing an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to the second embodiment of the present disclosure.
The abnormality warning device 100B includes an abnormality determination device 1000B and a warning output unit 2000.

The abnormality determination device 1000B obtains an image, and uses the image to output the state of the determination target captured in the image.
The abnormality determination device 1000B shown in FIG. 4 includes an image acquisition unit 1100B, a feature extraction unit 1300B, and an abnormal state determination unit 1500B.
Here, in abnormality determination device 1000B, the neural network unit already described is configured to include, as will be described later, a first neural network unit 1320 and a second neural network unit 1520. First neural network unit 1320 is included in feature extraction unit 1300B, and second neural network unit 1520 is included in abnormal state determination unit 1500B.
In addition, in the abnormality determination device 1000B, the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300B, and the second output storage unit is included in the abnormal state determination unit 1500B.
In addition, in abnormality determination device 1000B, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. First state comparison unit 1340 is included in feature extraction unit 1300B, and second state comparison unit 1540 is included in abnormal state determination unit 1500B.
In addition, in the abnormality determination device 1000B, the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580. The first process control unit 1380 is included in the feature extraction unit 1300B, and the second process control unit 1580 is included in the abnormal state determination unit 1500B.

Since the image acquisition unit 1100B is similar to the image acquisition unit 1100A already described, a detailed description of the image acquisition unit 1100B will be omitted here.

The feature extraction unit 1300B uses the image to output a feature map that represents the characteristic state of the subject contained in the image. For example, the feature extraction unit 1300B extracts parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents the state of the subject that is characteristic of an abnormal state such as drowsiness.

An example of the internal configuration of the feature extraction unit 1300B will be described.
FIG. 5 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the feature extraction unit 1300B in the abnormality determination device 1000B.
The feature extraction unit 1300B shown in FIG. 5 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340B), a processing control unit (first processing control unit 1380B), and an output storage unit (first output storage unit 1390).

The first neural network unit 1320 has multiple layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be judged contained in the image.
The first neural network unit 1320 shown in FIG. 5 includes an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combination unit 1328.

The image branching unit 1321 branches and inputs an image to multiple nodes in a layer of a neural network. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 5, the image is branched into two.

The convolution layer unit 1322 performs filtering to extract characteristic parts of the object to be determined in the image.
The convolution layer unit 1322 shown in FIG. 5 includes a first convolution layer 1323 and a second convolution layer 1324.
The first convolutional layer 1323 and the second convolutional layer 1324 perform filtering to extract face (body) parts for detecting a drowsy state, for example, by convolution processing (cross-correlation processing) using pre-prepared convolution filters of size 3x3 or 5x5.

The pooling layer unit 1325 shown in FIG. 5 includes a first pooling layer 1326 and a second pooling layer 1327 .
The first pooling layer 1326 and the second pooling layer 1327 generate an image relating to features that are robust against image position, for example, by calculating the maximum or average value for each predetermined region.

The first neural network unit 1320 shown in FIG. 5 includes two pairs of convolutional layers and pooling layers, but it is also effective to include one pair or three or more pairs.
In addition, the first pooling layer and the second pooling layer 1327 may be followed by two or more convolution layers and two or more pooling layers.
In addition, a normalized linear unit layer (activation function) or the like (not shown) may be included after the convolution layer.

The image combination unit 1328 combines multiple images output through the convolution layer and the pooling layer.
The image combining unit 1328 extracts areas that indicate signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map.

The first state comparison unit 1340B prestores a reference value, which is the standard for the input value to each layer in the first neural network unit 1320, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.

The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the layer. In the explanation of the processing described later, a case in which the previous value is used as the reference value will be explained.
In this case, the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but in the second and subsequent processing (processing for the second image), it determines changes in the state of the object to be judged.
However, the following values may be stored and used as the reference values.

The reference value can use typical processing results that have been learned and modeled in advance.
In other words, the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
By using typical processing results, comparisons can be made without being influenced by noise or anomalies contained in the actual observed images.

In addition, the reference value may use a part of the results determined based on advance information.
The partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head.
Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network.
This allows for reduced memory usage and data handling, resulting in faster processing speeds.

The first state comparison section 1340 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image.
More specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value.

The first state comparison section 1340B shown in FIG.
In the image state comparison unit 1341, "storing" means holding the values of the two-dimensional images (feature maps) of the respective input sources.
In addition, in the image state comparison unit 1341, comparison means, for example, taking the absolute value of the difference for each element (pixel) between a stored two-dimensional image (from the previous time/previous frame) and the latest two-dimensional image, and then comparing the sum or average with a predetermined threshold value.

The image state comparison unit 1341 shown in FIG. 5 includes a storage unit 1341a and a comparison processing unit 1341b.

When the first state comparison unit 1340 determines that the state of the object to be determined has not changed, the first processing control unit 1380B instructs the first neural network unit 1320 not to execute processing in the second layer and subsequent layers of the object to be determined, and also instructs the first processing control unit 1380B to output the feature map stored in the first output storage unit 1390.
The first process control unit 1380B shown in FIG. 15 includes a feature extraction process control unit 1381B.
The feature extraction process control unit 1381B executes the function of the first process control unit 1380B in the feature extraction unit 1300.

The first output storage section 1390 stores the output data (decision value (probability value)) output by the first neural network section 1320 .
The first output storage unit 1390 stores the feature map output by the first neural network unit 1320 .
The first output storage unit 1390 shown in FIG.
The combined image storage unit 1391 stores a feature map, which is a combined image combined and output by the first neural network unit 1320 .

An example of the internal configuration of the abnormal state determination unit 1500B will be described.
FIG. 6 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the abnormality state determination unit 1500B in the abnormality determination device 1000B.
The abnormal condition determination unit 1500B uses a feature map, which is a two-dimensional image, to output a determination value indicating the condition of the object to be determined.
The abnormal state determination unit 1500B shown in FIG. 6 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.

The neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data.
The neural network unit (second neural network unit 1520 ) shown in FIG. 6 includes a state classification unit 1525 and a probability output layer 1529 .

The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector, and further consolidating the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing/not dozing, eyeballs: dozing/not dozing).
The state classification unit 1525 shown in FIG. 6 includes a first fully connected layer 1527 and a second fully connected layer 1528.
The state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through a first fully connected layer 1527 and a second fully connected layer 1528 .

The probability output layer 1529 applies a softmax function, for example, and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.

The second neural network unit 1520 shown in FIG. 6 has two fully connected layers, but it may be configured to have three or more fully connected layers if there are no fully connected layers.

The second state comparison unit 1540B prestores a reference value that is a standard for the input value to the third layer, which is at least one of the layers in the second neural network unit 1520, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the third layer, to determine a change in the state of the object to be determined.

The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer. In the explanation of the processing described later, a case in which the previous value is used as the reference value will be explained.
In this case, the second state comparison unit 1540B does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
However, the following values may be stored and used as the reference values.

The reference value can use typical processing results that have been learned and modeled in advance.
In other words, the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
By using typical processing results, comparisons can be made without being influenced by noise or anomalies contained in the actual observed images.

In addition, the reference value may use a part of the results determined based on advance information.
The partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head.
Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network.
This allows for reduced memory usage and data handling, resulting in faster processing speeds.

The second state comparison section 1540B uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, second state comparison section 1540B determines that the state of the determination target is unchanged.
More specifically, when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value, the second state comparing section 1540B determines that the state of the determination target is unchanged.

The second state comparison unit 1540B shown in FIG. 6 includes a first fully connected layer state comparison unit 1541.
In the first fully connected layer state comparison unit 1541, "storing" means holding the values of the one-dimensional vectors of the respective input sources.
In addition, in the first fully connected layer state comparison unit 1541, comparison means, for example, taking the absolute value of the difference for each element between the stored one-dimensional vector (of the previous time/previous frame) and the latest one-dimensional vector, and then comparing the sum or average with a predetermined threshold value.

The first fully connected layer state comparison unit 1541 shown in FIG. 6 includes a storage unit 1541a and a comparison processing unit 1541b.
The storage unit 1541 a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer 1528 , for each output by the first fully connected layer 1527 .
The comparison processing unit 1541b compares the current value with a reference value.

When the second state comparison unit 1540 determines that the state of the object to be judged has not changed, the second processing control unit 1580B instructs the second neural network unit 1520 not to execute processing in the layer of the object to be judged and subsequent layers, and also instructs the second processing control unit 1580B to output the judgment value stored in the second output storage unit as output data.
The second process control unit 1580B shown in FIG. 6 includes an abnormal state determination process control unit 1581B.
The abnormal state determination process control unit 1581B functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500B.

The second output storage unit 1590 stores the decision value output by the second neural network unit 1520 .
The output storage section (second output storage section) 1590 shown in FIG.

The probability storage unit stores the judgment value output by the second neural network unit 1520 .
The judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.

Returning to the explanation of FIG.
The warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.

The abnormality determination device 1000B shown in FIG. 4 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000B is equivalent to the abnormality warning device 100B shown in FIG. 4. In the following explanation, the abnormality determination device 1000B will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100B and the abnormality determination device 1000B.

In addition to the above configuration, abnormality determination device 1000B may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
A control unit (not shown) controls the entire abnormality determination device 1000B and each of its components. The control unit (not shown) starts up the abnormality determination device 1000A in accordance with, for example, an external command. The control unit (not shown) also controls the state (operation state = start-up, shutdown, sleep, etc.) of the abnormality determination device 1000B.
A storage unit (not shown) stores each piece of data used in the abnormality determination device 1000B. The storage unit (not shown) stores, for example, outputs (output data) from each component in the abnormality determination device 1000B, and outputs data requested by each component to the component that has made the request.
The communication unit (not shown) communicates with an external device. For example, the communication unit communicates between the abnormality determination device 1000B and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000B does not have a display unit or an audio output unit, the communication unit communicates between the abnormality determination device 1000B and an external device such as a display unit or an audio output device.

An example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment will be described.
FIG. 7 is a flowchart showing an example of processing by the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
When an image is input from a camera, for example, the abnormality determination device 1000B starts the process shown in FIG.

Abnormality determination device 1000B executes image acquisition processing (step ST2100).
In the image acquisition process, the image acquisition unit 1100 of the abnormality determination device 1000B acquires and outputs an image.

Abnormality determination device 1000B executes a storage and state comparison process (step ST2200).
In the state storage and state comparison process, the first state comparison unit 1340B of the abnormality determination device 1000B stores the previous value (reference value), which is the input value to the layer, for each layer of the first neural network unit 1320 for at least the first time after the process starts.
The first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value (previous value = reference value), but for the second and subsequent processing (processing for the second image), it determines changes in the state of the object to be judged.

Abnormality determination device 1000B executes a feature extraction process control process (step ST2300).
The feature extraction process control unit 1381 of the abnormality determination device 1000B executes a normal process command or an omission process command to the first neural network unit 1320 depending on the determination result by the first state comparison unit 1340.
Furthermore, when issuing an omission processing command, the feature extraction processing control unit 1381 issues an output command to the combined image storage unit 1391 .

Abnormality determination device 1000B executes combined image output processing (step ST2400).
The first neural network unit 1320 executes normal processing and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image, whereby the feature extraction unit 1300 outputs a combined image.

Abnormality determination device 1000B executes a process of storing the fully connected layer states and comparing the fully connected layer states (step ST2500).
The second state comparison unit 1540 in the abnormality determination device 1000B pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is to be newly input to that layer.
The second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).

Abnormality determination device 1000B executes an abnormal state determination process control process (step ST2600).
The abnormal state determination process control unit 1581B issues a normal command or an omission command to the second neural network unit 1520. When issuing an omission command, the abnormal state determination process control unit 1581B issues an output command to the probability storage unit 1591.

Abnormality determination device 1000B executes a result output process (step ST2700).
The second neural network unit 1520 executes normal processing and outputs a judgment value, or the probability storage unit 1591 outputs a previous judgment value (probability value), so that the abnormal state judgment unit 1500 outputs the judgment value as output data.

Abnormality determination device 1000B executes an alarm output process (step ST2800).
In the alarm output process, the alarm output unit 2000 of the abnormality determination device 1000B acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc. The alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.

When abnormality determining device 1000B executes the process of step ST3800, it ends the series of processes shown in FIG. 7 and repeats the process from step ST3100.
In addition, the abnormality determination device 1000B is also turned off in conjunction with, for example, the camera being turned off.

Here, a detailed example of the processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment will be described.
FIG. 8 is a flowchart showing an example of more detailed processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
The flowchart in FIG. 8 shows an example of processing equivalent to the processing from step ST2200 to step ST2700 in the flowchart in FIG.
When the feature extraction unit 1300B in the abnormality determination device 1000B starts the process of step ST2200, the feature extraction unit 1300B first executes a storage process (step ST2201). In the storage process, the state comparison unit 1340 of the feature extraction unit 1300B stores a previous value that is an input value of the first layer, which is at least one layer among all layers of the neural network unit 1320 (first neural network unit 1320).
Furthermore, the state comparison unit 1340 stores the input values of the first layer each time processing is performed in the first layer.

Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes a previous storage determination process (step ST2202). In the previous storage determination process, the state comparison unit 1340 of the feature extraction unit 1300B determines whether the previous input value (previous value = reference value) is stored.

The feature extraction unit 1300B in the abnormality determination device 1000B executes a comparison process (step ST2203). In the comparison process, the state comparison unit 1340 of the feature extraction unit 1300B calculates the difference (differential value) between the current value, which is the value that is about to be newly input to the first layer, and the previous value (reference value).

Feature extraction unit 1300B in abnormality determination device 1000B executes a difference determination process (step ST2204). In the difference determination process, state comparison unit 1340 of feature extraction unit 1300B uses the difference value to determine a change in the state of the determination target.
Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison section 1340 determines that the state of the determination target is unchanged.
More specifically, the state comparison unit 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.

If the state comparison unit 1340 of the feature extraction unit 1300B determines that the data was not previously stored (step ST2202 "NO"), or if the state comparison unit 1340 determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed (step ST2204 "NO"), the feature extraction unit 1300B in the abnormality determination device 1000B executes a normal processing command (step ST2301). In the normal processing command, the processing control unit 1380 of the feature extraction unit 1300B outputs a command to execute processing in the first layer of the neural network unit 1320 (first neural network unit 1320).

Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes output processing (step ST2401). In the output processing, the first neural network unit 1320 of the feature extraction unit 1300B executes processing according to the normal processing command, and outputs output data indicating the determination result.

Next, the feature extraction unit 1300B in the abnormality determination device 1000B executes an output storage process (step ST2402). In the output storage process, the output storage unit 1390 (first output storage unit 1390) of the feature extraction unit 1300B stores the output data output by the neural network unit 1320.

If the state comparison unit 1340 determines that the difference is less than the threshold value and that the state of the object to be determined has not changed (step ST2204 "YES"), the feature extraction unit 1300B in the abnormality determination device 1000B executes an omission processing command (step ST2302). In the omission processing command, the processing control unit 1380 of the feature extraction unit 1300B instructs the neural network unit 1320 not to execute processing in the first layer or subsequent layers, which is at least one layer out of the multiple layers, and instructs the output storage unit 1390 to output the output data stored therein.

The feature extraction unit 1300B in the abnormality determination device 1000B executes a stored data output process (step ST2403). In the stored data output process, when the output storage unit 1390 (combined image storage unit 1391 of the output storage unit 1390) of the feature extraction unit 1300B receives a command from the process control unit 1380, it outputs the stored output data, that is, the combined image, to the alarm output unit 2000.

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes a fully connected layer output storage process (step ST2501). In the fully connected layer output storage process, the state comparison unit 1540 of the abnormal state determination unit 1500B stores the value output from the fully connected layer (the first fully connected layer 1527 or the second fully connected layer 1528) in the state classification unit 1525 as the state of the fully connected layer.

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes a previous storage determination process (step ST2502). In the previous storage determination process, the state comparison unit 1540 of the abnormal state determination unit 1500B determines whether the previous input value (previous value = reference value) is stored.

If the abnormal state determination unit 1500B in the abnormality determination device 1000B determines that a previous value has been stored ("YES" in step ST2502), it executes a comparison process (step ST2503). In the comparison process, the state comparison unit 1540 of the abnormal state determination unit 1500B calculates the difference (differential value) between the current value, which is the value that is about to be newly input to the first layer (the second fully connected layer 1528 or the probability output layer 1529), and the previous value (reference value).

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes a difference determination process (step ST2504). In the difference determination process, the state comparison unit 1540 of the abnormal state determination unit 1500B determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the previous value (reference value) and a pre-stored threshold value.

If the state comparison unit 1540 of the abnormal state determination unit 1500B determines that the difference is equal to or greater than the threshold value (step ST2504 "NO"), the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a normal processing command (step ST2601). In the normal processing command, the processing control unit 1580 of the abnormal state determination unit 1500B executes a normal processing command to the neural network unit 1520 (second neural network unit 1520).

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes an output process (step ST2701). In the output process, the neural network unit 1520 (second neural network unit 1520) of the abnormal state determination unit 1500B outputs the determination value as output data.

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes an output storage process (step ST2702). In the output storage process, the output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500B stores the determination value output by the neural network unit 1520 (second neural network unit 1520).

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes an omission processing command (step ST2602). In the omission processing command, the processing control unit 1580 of the abnormal state determination unit 1500B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be determined, and instructs the second output storage unit to output the determination value stored therein.

The abnormal state determination unit 1500B in the abnormality determination device 1000B executes a stored data output process (step ST2703). In the stored data output process, the second output storage unit 1590 of the abnormal state determination unit 1500B stores the determination value output by the neural network unit 1520 (second neural network unit 1520).

When the second neural network unit 1520 or the second output storage unit 1590 outputs the judgment value, the series of processes shown in FIG. 21 ends.

In the figure, an example of a configuration is shown in which the first state comparison unit 1340B has an image state comparison unit 1341, and the second state comparison unit 1540B has a first fully connected layer state comparison unit 1541. However, the configuration may be one or a combination of the following: the image state comparison unit 1341, a convolution layer state comparison unit that determines a state change in the convolution layer (see the image state comparison unit 1342 in the embodiment described later), a pooling layer state comparison unit that determines a state change in the pooling layer (see the pooling layer state comparison unit 1343 in the embodiment described later), a first fully connected layer state comparison unit (see the first fully connected layer state comparison unit 1541 in the embodiment described later), and a second fully connected layer state comparison unit that determines a state change in the second fully connected layer 1528 (see the second fully connected layer state comparison unit 1542 in the embodiment described later). For example, when any one of them is adopted, it can be realized by the processing shown in the following Figures 9, 10, 11, 12, and 13.

An example of the process of the feature extraction unit 1300B when the state comparison unit (first state comparison unit 1340B) is the image state comparison unit 1341 will be described.
FIG. 9 is a flowchart showing an example of a process of the feature extraction unit 1300B in the case where the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341.

When the feature extraction section 1300B starts the process, first, the first state comparison section 1340B of the feature extraction section 1300B acquires an image acquisition command and executes an image storage process (step ST2211).
In the image storage process, the image state comparison section 1341 of the first state comparison section 1340B stores the image that is the input data for the first neural network section 1320.
Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image (the image before being processed by the first neural network unit 1320) in the storage unit 1341a.

The image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST2212).
The image state comparison section 1341 refers to the storage section 1341a to determine whether or not the previously input image is stored.

When it is determined that the image has been stored previously ("YES" in step ST2212), the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST2213).
The comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.

The comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST2214).
The comparison processing unit 1341b uses the difference value between the image input this time and the image input last time and a pre-stored threshold value to determine the magnitude of the difference between a reference value (the image input last time) and a current value (the image to be input this time).

If the image state comparison unit 1341 determines that no value was stored last time (step ST2212 ``NO''), or if the image state comparison unit 1341 determines that the difference between the previous value (reference value) and the current value is greater than or equal to a threshold value (step ST2214 ``NO''), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2311).
First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2411).
The combined image storage section 1391, which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2412).

If the image state comparison unit 1341 determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value (step ST2214 "YES"), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2312) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2413).
In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
The combined image storage unit 1391 outputs the combined image in accordance with the output command.

Next, an example of the processing of the feature extraction unit 1300B in a case where the state comparison unit (first state comparison unit 1340B) is a convolution layer state comparison unit (see the convolution layer state comparison unit 1342 in the embodiment described later) will be described.
FIG. 10 is a flowchart illustrating an example of a process of the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a convolutional layer state comparison unit.

When the feature extraction unit 1300B starts the process, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes a convolution layer state storage process (step ST2221).
In the convolution layer state storage process, the convolution layer state comparison unit of the first state comparison unit 1340B stores an input value (a value before being processed by the pooling layer unit 1325) which is an output value of the convolution layer unit 1322 and is input data to the pooling layer unit 1325.
In addition, every time the convolution layer unit 1322 outputs an output value, the convolution layer state comparison unit stores the output value (the value before being processed by the pooling layer unit 1325) in the storage unit.

The convolution layer state comparison unit executes a process of determining whether the data has been stored previously (step ST2222).
The convolution layer state comparison unit refers to the storage unit and determines whether a previously input value (reference value: in this explanation, the previous value) is stored.

When the convolution layer state comparison unit determines that the previous state has been stored ("YES" in step ST2222), it executes a comparison process between the current state and the previous state (step ST2223).
The comparison processing unit of the convolution layer state comparison unit compares the value that is currently being input to the pooling layer (current value) with the value that was previously input (previous value=reference value).

The comparison processing unit executes a process of determining whether the difference is less than a threshold value (step ST2224).
The comparison processing unit determines the magnitude of the difference between a reference value (the value input previously) and the current value (the value being input currently) using a difference value between the value being input currently (the current value) and the value input previously (the previous value) and a pre-stored threshold value.

If the convolutional layer state comparison unit determines that no value was stored last time ("NO" in step ST2222), or if the convolutional layer state comparison unit determines that the difference between the previous value (reference value) and the current value is equal to or greater than a threshold value ("NO" in step ST2224), the feature extraction processing control unit 1381, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321).
First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2421).
The combined image storage section 1391, which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2422).

If the convolution layer state comparison unit determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value (step ST2224 "YES"), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423).
In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
The combined image storage unit 1391 outputs the combined image in accordance with the output command.

Next, an example of the process of the feature extraction unit 1300B in a case where the state comparison unit (first state comparison unit 1340B) is a pooling layer state comparison unit (see the pooling layer state comparison unit 1343 in the embodiment described later) will be described.
FIG. 11 is a flowchart illustrating an example of a process performed by the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a pooling layer state comparison unit.

When the feature extraction unit 1300B starts the process, first, the first state comparison unit 1340B of the feature extraction unit 1300B executes a pooling layer state storage process (step ST2221).
In the pooling layer state storage process, the pooling layer state comparison unit of the first state comparison unit 1340B stores the input value (the value before being processed by the image combination unit 1328), which is the output value of the pooling layer unit 1325 and is input data to the image combination unit 1328.
In addition, every time the pooling layer unit 1325 outputs an output value, the pooling layer state comparison unit stores the output value (the value before being processed by the image combination unit 1328) in the storage unit.

The pooling layer state comparison unit executes a process of determining whether or not the data has been stored previously (step ST2222).
The pooling layer state comparison unit refers to the storage unit and determines whether a previously input value (reference value: in this explanation, the previous value) is stored.

When it is determined that the data has been stored previously ("YES" in step ST2222), the pooling layer state comparison unit executes a process of comparing the current data with the previous data (step ST2223).
The comparison processing unit of the pooling layer state comparison unit compares the value currently being input to the image combination unit 1328 (current value) with the value previously input (previous value).

The comparison processing unit executes a process of determining whether the difference is less than a threshold value (step ST2224).
The comparison processing unit determines the magnitude of the difference between a reference value (the value input previously) and the current value (the value being input currently) using a difference value between the value being input currently (the current value) and the value input previously (the previous value) and a pre-stored threshold value.

If the pooling layer state comparison unit determines that no value was stored last time (step ST2222 ``NO''), or if the pooling layer state comparison unit determines that the difference between the previous value (reference value) and the current value is greater than or equal to a threshold value (step ST2224 ``NO''), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321).
First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2421).
The combined image storage section 1391, which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2422).

If the pooling layer state comparison unit determines that the difference between the previous value (reference value) and the current value is smaller than the threshold value (step ST2224 "YES"), the feature extraction processing control unit 1381B, which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423).
In accordance with the omission processing command, the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
The combined image storage unit 1391 outputs the combined image in accordance with the output command.

Next, an example of the process of the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) is the first layer state comparison unit 1541 will be described.
FIG. 12 is a flowchart showing an example of processing of the abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is the first layer state comparison unit 1541.

When the abnormal state determination unit 1500B executes processing on the data output by the feature extraction unit 1300B (step ST2511), the second state comparison unit 1540B first acquires the first fully connected layer output (step ST2512). It then executes a first fully connected layer state storage process (step ST2513). The first layer state comparison unit 1541 in the second state comparison unit 1540B stores the output value from the first fully connected layer 1527, which is the input value of the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).

The first layer state comparing unit 1541 executes a process of determining whether or not the data has been previously stored (step ST2514).
The first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) is stored.

If the first layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST2514), it executes a comparison process between the current value and the previous value (step ST2515). The first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).

The first layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST2516).
The comparison processing unit 1541b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value and a pre-stored threshold value.

If the comparison processing unit 1541b determines that no data was stored previously (step ST2514 "NO"), or if the first layer state comparison unit 1541 determines that the difference is equal to or greater than the threshold value (step ST2516 "NO"), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2611).
Second neural network unit 1520 executes normal processing in accordance with the normal processing command, and outputs a decision value (step ST2711).
Probability storage section 1591, which is second output storage section 1590, stores the decision value output by second neural network section 1520 (step ST2712).

If the first layer state comparator 1541 determines that the difference is smaller than the threshold value ("YES" in step ST2516), the abnormal state determination process controller 1581B, which is the second process controller 1580, executes an omission process command (step ST2612).
In the omission processing command, the abnormal state judgment processing control unit 1581B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.
In accordance with the omission processing command, the second neural network unit 1520 does not execute processing of the subsequent layers in the second neural network unit 1520 .
The probability storage section 1591, which is the second output storage section 1590, outputs the decision value in accordance with the output command (step ST2713).

Next, an example of the processing of the abnormal state determination unit 1500B when the state comparison unit (second state comparison unit 1540B) is a second fully connected layer state comparison unit (see the second fully connected layer state comparison unit 1542 in the embodiment described later) will be described.
FIG. 13 is a flowchart showing an example of processing by abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in abnormal state determination unit 1500B according to embodiment 2 of the present disclosure is the second layer state comparison unit.

After the abnormal state determination unit 1500B executes processing on the data output by the feature extraction unit 1300B (step ST2521), the second fully connected layer 1528 then executes processing on the output by the first fully connected layer (step ST2522).
Next, the second state comparison unit 1540B acquires the second fully connected layer output (step ST2523). Next, the second state comparison unit 1540B executes a second fully connected layer state storage process (step ST2524). The second layer state comparison unit in the second state comparison unit 1540B stores the output value from the second fully connected layer 1528 and the input value of the probability output layer 1529 (the value before being processed by the probability output layer 1529).

The second layer state comparing unit executes a process of determining whether the data has been stored previously (step ST2525).
The second layer state comparison unit refers to the storage unit and determines whether a reference value (previous value) is stored.

If the second layer state comparison unit determines that the value was stored last time ("YES" in step ST2525), it executes a comparison process between the current value and the previous value (step ST2526). The second layer state comparison unit compares the current value with the reference value (previous value).

The second layer state comparing unit executes a process of determining whether the difference is smaller than a threshold value (step ST2527).
The comparison processing section determines whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value (previous value) and a pre-stored threshold value.

If the comparison processing unit determines that no data was stored previously (step ST2525 "NO"), or if the second layer state comparison unit determines that the difference is greater than or equal to the threshold value (step ST2527 "NO"), the abnormal state determination processing control unit 1581B, which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2621).
Second neural network unit 1520 executes normal processing in accordance with the normal processing command, and outputs a determination value (step ST2721).
Probability storage section 1591, which is second output storage section 1590, stores the decision value output by second neural network section 1520 (step ST2722).

If the first fully connected layer state comparison unit 1541 judges that the difference is smaller than the threshold value (step ST2527 "YES"), the abnormal state judgment process control unit 1581B, which is the second process control unit 1580, executes an omission process command (step ST2622).
In the omission processing command, the abnormal state judgment processing control unit 1581B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.
In accordance with the omission processing command, the second neural network unit 1520 does not execute processing of the subsequent layers in the second neural network unit 1520 .
The probability storage section 1591, which is the second output storage section 1590, outputs a decision value in accordance with the output command (step ST2723).

The abnormality determination device of the present disclosure is further configured as follows.

"The neural network unit includes a first neural network unit and a second neural network unit,
The output storage unit includes a first output storage unit and a second output storage unit,
the state comparison unit includes a first state comparison unit and a second state comparison unit,
The processing control unit includes a first processing control unit and a second processing control unit,
the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
the first output storage unit that stores the feature map output by the first neural network unit;
the first state comparison unit pre-stores a reference value that is a reference for an input value to a second layer that is at least one of the layers in the first neural network unit, and judges a change in the state of the object to be judged by using a difference value between the reference value and a current value that is a value that is to be newly input to the second layer;
and,
the first processing control unit instructs the first neural network unit not to execute processing in the second layer and subsequent layers of the object to be determined when the first state comparison unit determines that the state of the object to be determined has not changed, and instructs the first processing control unit to output the feature map stored in the first output storage unit;
A feature extraction unit including:
the second neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring the feature map output by the feature extraction unit, and outputting a judgment value indicating a state of the object to be judged as the output data using the feature map;
the second output storage unit for storing the judgment value output by the second neural network unit;
the second state comparison unit stores in advance a reference value that is a reference for an input value to a third layer that is at least one of the layers in the second neural network unit, and judges a change in the state of the object to be judged by using a difference value between the reference value and a current value that is a value that is to be newly input to the third layer;
and,
the second processing control unit instructs the second neural network unit not to execute processing in the third layer and subsequent layers of the object to be judged when the second state comparison unit judges that the state of the object to be judged has not changed, and instructs the second processing control unit to output the judgment value stored in the second output storage unit as the output data;
An abnormality state determination unit configured to include
An abnormality determination device equipped with the above.

As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The first state comparison unit and the second state comparison unit are
a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value;
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.

"The first state comparison unit and the second state comparison unit are
When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

The abnormality determination device of the present disclosure is further configured as follows.
"The first state comparison unit and the second state comparison unit are
When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
An abnormality determination device characterized by the above.

As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

Embodiment 3.
In the third embodiment, a form in which the basic mechanism of the present disclosure is applied to all layers in a neural network will be described.
In the third embodiment, the configuration and processing already described will be omitted as appropriate.

An example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment will be described.
FIG. 14 is a diagram showing an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment of the present disclosure.
The abnormality warning device 100C includes an abnormality determination device 1000C and a warning output unit 2000.

The abnormality determination device 1000C obtains an image, and uses the image to output the state of the object to be determined that is captured in the image.
An abnormality determination device 1000C shown in FIG. 14 includes an image acquisition unit 1100C, a feature extraction unit 1300C, and an abnormal state determination unit 1500C.
Here, in abnormality determination device 1000C, the neural network unit already described is configured to include, as will be described later, a first neural network unit 1320 and a second neural network unit 1520. First neural network unit 1320 is included in feature extraction unit 1300C, and second neural network unit 1520 is included in abnormal state determination unit 1500C.
In addition, in the abnormality determination device 1000C, the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit. The first output storage unit 1390 is included in the feature extraction unit 1300C, and the second output storage unit is included in the abnormal state determination unit 1500C.
Moreover, in abnormality determination device 1000C, the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540. First state comparison unit 1340 is included in feature extraction unit 1300C, and second state comparison unit 1540 is included in abnormal state determination unit 1500C.
Moreover, in the abnormality determination device 1000C, the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580. The first process control unit 1380 is included in the feature extraction unit 1300C, and the second process control unit 1580 is included in the abnormal state determination unit 1500C.

Since the image acquisition unit 1100C is similar to the image acquisition units 1100A and 1100B already described, a detailed description of the image acquisition unit 1100C will be omitted here.

The feature extraction unit 1300C uses the image to output a feature map that represents the characteristic state of the subject contained in the image. For example, the feature extraction unit 1300C extracts parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents the state of the subject that is characteristic of an abnormal state such as drowsiness.

An example of the internal configuration of the feature extraction unit 1300C will be described.
FIG. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C.
The feature extraction unit 1300C shown in FIG. 15 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340C), a processing control unit (first processing control unit 1380C), and an output storage unit (first output storage unit 1390).

The first neural network unit 1320 has multiple layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be judged contained in the image.
The first neural network unit 1320 shown in FIG. 15 includes an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combination unit 1328.

The image branching unit 1321 branches and inputs an image to multiple nodes in a layer of a neural network. The image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 15, the image is branched into two.

The convolution layer unit 1322 performs filtering to extract characteristic parts of the object to be determined in the image.
The convolution layer unit 1322 shown in FIG. 15 includes a first convolution layer 1323 and a second convolution layer 1324.
The first convolutional layer 1323 and the second convolutional layer 1324 perform filtering to extract face (body) parts for detecting a drowsy state, for example, by convolution processing (cross-correlation processing) using pre-prepared convolution filters of size 3x3 or 5x5.

The pooling layer unit 1325 shown in FIG. 15 includes a first pooling layer 1326 and a second pooling layer 1327 .
The first pooling layer 1326 and the second pooling layer 1327 generate an image relating to features that are robust against image position, for example, by calculating the maximum or average value for each predetermined region.

The first neural network unit 1320 shown in FIG. 15 includes two pairs of convolutional layers and pooling layers, but it is also effective to include one pair or three or more pairs.
In addition, the first pooling layer and the second pooling layer 1327 may be followed by two or more convolution layers and two or more pooling layers.
In addition, a normalized linear unit layer (activation function) or the like (not shown) may be included after the convolution layer.

The image combination unit 1328 combines multiple images output through the convolution layer and the pooling layer.
The image combining unit 1328 extracts areas that indicate signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map.

The first state comparison unit 1340C prestores a reference value, which is the standard for the input value to each layer in the first neural network unit 1320, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.

The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the layer. In the explanation of the processing described later, a case in which the previous value is used as the reference value will be explained.
In this case, the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but in the second and subsequent processing (processing for the second image), it determines changes in the state of the object to be judged.
However, the following values may be stored and used as the reference values.

The reference value can use typical processing results that have been learned and modeled in advance.
In other words, the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
By using typical processing results, comparisons can be made without being influenced by noise or anomalies contained in the actual observed images.

In addition, the reference value may use a part of the results determined based on advance information.
The partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head.
Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network.
This allows for reduced memory usage and data handling, resulting in faster processing speeds.

The first state comparison section 1340 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image.
More specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value.

The first state comparison unit 1340C shown in FIG. 15 includes an image state comparison unit 1341, a convolution layer state comparison unit 1342, and a pooling layer state comparison unit 1343.
In the image state comparison unit 1341, the convolution layer state comparison unit 1342, and the pooling layer state comparison unit 1343, "storing" means holding the values of the two-dimensional images (feature maps) that are the respective input sources.
Furthermore, in the image state comparison unit 1341, the convolution layer state comparison unit 1342, and the pooling layer state comparison unit 1343, comparison means, for example, taking the absolute value of the difference for each element (pixel) between a stored two-dimensional image (of the previous time/frame) and the latest two-dimensional image, and further comparing the sum or average with a predetermined threshold value.

The image state comparison unit 1341 shown in FIG. 15 includes a storage unit 1341a and a comparison processing unit 1341b.

The convolutional layer state comparison unit 1342 shown in FIG. 15 includes a storage unit 1342a and a comparison processing unit 1342b.

The pooling layer state comparison unit 1343 shown in FIG. 15 includes a storage unit 1343a and a comparison processing unit 1343b.

When the first state comparison unit 1340 determines that the state of the object to be judged has not changed, the first processing control unit 1380C instructs the first neural network unit 1320 not to execute processing in layers subsequent to the layer of the object to be judged, and also instructs the first output storage unit 1390 to output the feature map stored in the first output storage unit 1390.
The first process control unit 1380C shown in FIG. 15 includes a feature extraction process control unit 1381C.
The feature extraction process control unit 1381C executes the function of the first process control unit 1380C in the feature extraction unit 1300.

The first output storage section 1390 stores the output data (decision value (probability value)) output by the first neural network section 1320 .
The first output storage unit 1390 stores the feature map output by the first neural network unit 1320 .
The first output storage unit 1390 shown in FIG.
The combined image storage unit 1391 stores a feature map, which is a combined image combined and output by the first neural network unit 1320 .

Returning to the explanation of FIG.
The abnormal condition determination unit 1500C uses a feature map, which is a two-dimensional image, to output a determination value indicating the condition of the object to be determined.

An example of the internal configuration of the abnormal state determination unit 1500C will be described.
FIG. 16 is a diagram showing an example of the internal configuration of the abnormality warning device 100C and the abnormality state determination unit 1500C in the abnormality determination device 1000C.
The abnormal state determination unit 1500C shown in FIG. 16 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.

The neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data.
The neural network unit (second neural network unit 1520 ) shown in FIG. 16 includes a state classification unit 1525 and a probability output layer 1529 .

The state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector, and further consolidating the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing/not dozing, eyeballs: dozing/not dozing).
The state classification unit 1525 shown in FIG. 16 includes a first fully connected layer 1527 and a second fully connected layer 1528.
The state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through a first fully connected layer 1527 and a second fully connected layer 1528 .

The probability output layer 1529 applies a softmax function, for example, and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.

The second neural network unit 1520 shown in FIG. 16 has two fully connected layers, but it may be configured to have three or more fully connected layers if there are no fully connected layers.

The second state comparison unit 1540 prestores a reference value, which is the standard for the input value to each layer of the second neural network unit 1520, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.

The reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer. In the explanation of the processing to be described later, the previous value is used as the reference value.
In this case, the second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
However, the following values may be stored and used as the reference values.

The reference value can use typical processing results that have been learned and modeled in advance.
In other words, the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
By using typical processing results, comparisons can be made without being influenced by noise or anomalies contained in the actual observed images.

In addition, the reference value may use a part of the results determined based on advance information.
The partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head.
Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
In other words, the reference value may be the output value of a predetermined node among the nodes included in the neural network.
This allows for reduced memory usage and data handling, resulting in faster processing speeds.

The second state comparison section 1540 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result.
Specifically, when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, second state comparing section 1540 determines that the state of the determination target is unchanged.
More specifically, when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value, the second state comparing section 1540 determines that the state of the determination target is unchanged.

16 includes a first layer state comparison unit 1541 and a second layer state comparison unit 1542. The first layer state comparison unit 1541 is also referred to as a first fully connected layer state comparison unit 1541. The second layer state comparison unit 1542 is also referred to as a second fully connected layer state comparison unit 1542.
In the first fully connected layer state comparison unit 1541 and the second fully connected layer state comparison unit 1542, "storing" means holding the values of the one-dimensional vectors of the respective input sources.
In the first fully connected layer state comparison unit 1541 and the second fully connected layer state comparison unit 1542, comparison means, for example, taking the absolute value of the difference between a stored one-dimensional vector (of the previous time/frame) and the latest one-dimensional vector for each element, and then comparing the sum or average with a predetermined threshold value.

The first fully connected layer state comparison unit 1541 shown in FIG. 16 includes a storage unit 1541a and a comparison processing unit 1541b.
The storage unit 1541a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer, for each output by the first fully connected layer 1527.
The comparison processing unit 1541b compares the current value with a reference value.

The second fully connected layer state comparison unit 1542 shown in FIG. 16 includes a storage unit 1542a and a comparison processing unit 1542b.
The storage unit 1542a stores a reference value, which is an output value of the second fully connected layer and an input value of the probability output layer 1529, for each output by the second fully connected layer.
The comparison processing unit 1542b compares the current value with a reference value.

When the second state comparison unit 1540 determines that the state of the object to be judged has not changed, the second processing control unit 1580C instructs the second neural network unit 1520 not to execute processing in the layer of the object to be judged and subsequent layers, and also instructs the second processing control unit 1580C to output the judgment value stored in the second output storage unit as output data.
The second process control unit 1580C shown in FIG. 16 includes an abnormal state determination process control unit 1581C.
The abnormal state determination processing control unit 1581C functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500C.

The second output storage unit 1590 stores the decision value output by the second neural network unit 1520 .
The output storage section (second output storage section) 1590 shown in FIG.

The probability storage unit 1591 stores the judgment value output by the second neural network unit 1520 .
The judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.

Returning to the explanation of FIG.
The warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.

The abnormality determination device 1000C shown in FIG. 14 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000C is equivalent to the abnormality warning device 100C shown in FIG. 14. In the following explanation, the abnormality determination device 1000C will be explained as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100C and the abnormality determination device 1000C.

In addition to the above configuration, abnormality determination device 1000C may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
A control unit (not shown) controls the entire abnormality determination device 1000C and each of its components. The control unit (not shown) starts up the abnormality determination device 1000C in response to, for example, an external command. The control unit (not shown) also controls the state (operation state = start-up, shutdown, sleep, etc.) of the abnormality determination device 1000C.
A storage unit (not shown) stores each piece of data used in the abnormality determination device 1000C. The storage unit (not shown) stores, for example, outputs (output data) from each component in the abnormality determination device 1000C, and outputs data requested by each component to the component that has made the request.
The communication unit (not shown) communicates with an external device. For example, the communication unit communicates between the abnormality determination device 1000C and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000C does not have a display unit or an audio output unit, the communication unit communicates between the abnormality determination device 1000C and an external device such as a display unit or an audio output device.

An example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment will be described.
FIG. 17 is a flowchart showing an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
The abnormality determination device 1000C starts the process shown in FIG. 17 when an image is input from a camera, for example.

Abnormality determination device 1000C executes image acquisition processing (step ST3100).
In the image acquisition process, the image acquisition unit 1100 of the abnormality determination device 1000C acquires and outputs an image.

Next, abnormality determination device 1000C executes a storage and state comparison process (step ST3200).
In the state storage and state comparison process, the first state comparison unit 1340C of the abnormality determination device 1000C stores the previous values, which are the input values for each layer, which are the processing results for all layers of the first neural network unit 1320 for at least the first time after the start of processing.
The first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but in the second and subsequent processing (processing for the second image), it determines whether there has been a change in the state of the object to be judged.

Next, abnormality determination device 1000C executes feature extraction process control (step ST3300).
The feature extraction process control unit 1381 of the abnormality determination device 1000C executes a normal process command or an omission process command to the first neural network unit 1320 depending on the determination result by the first state comparison unit 1340.
Furthermore, when issuing an omission processing command, the feature extraction processing control unit 1381 issues an output command to the combined image storage unit 1391 .

Next, abnormality determination device 1000C executes combined image output processing (step ST3400).
The first neural network unit 1320 executes normal processing and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image, whereby the feature extraction unit 1300 outputs a combined image.

Abnormality determination device 1000C executes a process of storing the fully connected layer states and comparing the fully connected layer states (step ST3500).
The second state comparison unit 1540 in the abnormality determination device 1000C pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is to be newly input to that layer.
The second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).

Abnormality determination device 1000C executes an abnormal state determination process control process (step ST3600).
The abnormal state determination process control unit 1581C issues a normal command or an omission command to the second neural network unit 1520. When issuing an omission command, the abnormal state determination process control unit 1581C issues an output command to the probability storage unit 1591.

Abnormality determination device 1000C executes a result output process (step ST3700).
The second neural network unit 1520 executes normal processing and outputs a judgment value, or the probability storage unit outputs a previous judgment value (probability value), which causes the abnormality judgment unit to output the judgment value as output data.

Abnormality determination device 1000C executes an alarm output process (step ST3800).
In the alarm output process, the alarm output unit 2000 of the abnormality determination device 1000C acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc. The alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.

When abnormality determining device 1000C executes the process of step ST3800, it ends the series of processes shown in FIG. 17 and repeats the process from step ST3100.
In addition, the abnormality determination device 1000C is also turned off in conjunction with, for example, the camera being turned off.

A detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment will be described.
FIG. 18 is a flowchart showing a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.

When the feature extraction section 1300C starts the process, first, the first state comparison section 1340C of the feature extraction section 1300C executes an image storage process (step ST3210).
In the image storage process, the image state comparison section 1341 of the first state comparison section 1340C stores the image that is the input data for the first neural network section 1320.
Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.

The image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST3211).
The image state comparison section 1341 refers to the storage section 1341a to determine whether the previously input image is stored therein.

When it is determined that the image has been stored previously ("YES" in step ST3211), the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST3212).
The comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.

The comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST3213).
The comparison processing unit 1341b judges the magnitude of the difference between the reference value and the current value using a difference value between the currently input image and the previously input image and a pre-stored threshold value.

When the image state comparison unit 1341 judges that the difference between the previous value and the current value is greater than the threshold value ("NO" in step ST3213), the convolution layer state comparison unit 1342 executes a convolution layer state storage process (step ST3214).
In the convolution layer state storage process, the convolution layer state comparison unit 1342 stores input data for the convolution layer in the storage unit 1342a.

The convolution layer state comparison unit 1342 executes a process of determining whether or not the data has been previously stored (step ST3215).
The convolution layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value, which is a value input previously, is stored.

If the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that the previous state has been stored (step ST3215 "YES"), it executes a comparison process between the current state and the previous state (step ST3216).

The feature extraction section 1300C executes a process of determining whether the difference is less than a threshold value (step ST3217).
The comparison processing unit 1342b judges whether the difference between the previous value and the current value is large or small, using a difference value between the currently input image and the previously input image and a pre-stored threshold value.

If the comparison processing unit 1342b of the convolution layer state comparing unit 1342 determines that the difference between the previous value and the current value is greater than the threshold value ("NO" in step ST3217), the pooling layer state comparing unit 1343 executes a pooling layer state storage process (step ST3218).
In the pooling layer state storage process, the pooling layer state comparison unit 1343 stores input data for the pooling layer in the storage unit 1343a.

The comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process to determine whether the data was previously stored (step ST3219).

If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the value was stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).

The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process to determine whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).

If the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolution layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determine that the difference between the reference value (previous value) and the current value is greater than the threshold value (step ST3213 "NO", step ST3217 "NO", step ST3221 "NO"), the feature extraction unit 1300C executes normal processing command processing (step ST3311).
In the normal processing command process, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320 .

The feature extraction section 1300C executes an output process (step ST3411).
The first neural network portion 1320 of the feature extraction portion 1300C outputs the combined image.

The feature extraction section 1300C executes an output storage process (step ST3412).
The combined image storage section 1391 of the output storage section in the feature extraction section 1300C stores the combined image output from the first neural network section 1320.

If any of the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolution layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the previous value and the current value is smaller than the threshold value (step ST3213 "YES", step ST3217 "YES", step ST3221 "YES"), the feature extraction unit 1300C executes an omission processing command processing (step ST3312).
In the omission processing command processing, the feature extraction processing control unit 1381 commands the first neural network unit 1320 not to execute processing in layers subsequent to the layer to be judged, and also commands the first output storage unit 1390 to output the feature map stored therein.

The feature extraction section 1300C executes a stored data output process (step ST3413).
The feature extraction unit 1300C ends the process shown in FIG.

FIG. 19 is a flowchart showing a detailed first example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
The abnormal condition determination unit 1500C starts processing when it acquires the feature map output from the feature extraction unit 1300C. The abnormal condition determination unit 1500C first executes processing on the data output by the feature extraction unit 1300C (step ST3510).

The abnormal state determination unit 1500C acquires the fully connected layer output (step ST3511).
The second state comparison section 1540 in the abnormal state determination section 1500C obtains the output value output from the first fully connected layer 1527.

The first fully connected layer state comparison unit 1541 executes a first fully connected layer state storage process (step ST3512). The first fully connected layer state comparison unit 1541 stores the output value from the first fully connected layer 1527, which is the input value to the second fully connected layer 1528, as a reference value.

The first fully connected layer state comparing unit 1541 executes a process of judging whether the state has been stored previously (step ST3513).
The first fully connected layer state comparison unit 1541 refers to the storage unit 1541a to determine whether a reference value (previous value) is stored.

If the first fully connected layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST3513), it executes a comparison process between the current value and the previous value (step ST3514). The first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).

The first fully connected layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515).
The comparison processing unit 1541b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value and a pre-stored threshold value.

If the first fully connected layer state comparison unit 1541 determines that the difference is greater than the threshold value, normal processing is executed in the second fully connected layer 1528, and the second fully connected layer state comparison unit 1542 executes second fully connected layer state storage processing (step ST3516).
The second fully connected layer state comparison unit 1542 stores the output value from the second fully connected layer 1528, which is the input value to the probability output layer 1529, as a reference value.

The second fully connected layer state comparing unit 1542 executes a process of judging whether the state has been stored previously (step ST3517).
The second fully connected layer state comparison unit 1542 refers to the storage unit 1542a to determine whether a reference value (previous value) is stored.

If the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518). The second fully connected layer state comparison unit 1542 compares the current value with a reference value (previous value).

The second fully connected layer state comparing unit 1542 executes a process of determining whether the difference is smaller than the threshold value (step ST3519).
The comparison processing unit 1542b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value (previous value) and a pre-stored threshold value.

If the first fully connected layer state comparison unit 1541 determines that the difference is greater than the threshold value (step ST3515 "NO"), and if the second fully connected layer state comparison unit 1542 determines that the difference is greater than the threshold value (step ST3519 "NO"), the abnormal state determination unit 1500C executes normal processing command processing (step ST3611).

Abnormal state determination section 1500C executes output processing (step ST3711).
Neural network section 1520 (second neural network section 1520) in abnormal state determination section 1500C outputs a determination value.

Abnormal state determination section 1500C executes an output storage process (step ST3712).
An output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500C stores the determination value output by the neural network unit 1520 (second neural network unit 1520).

If the first fully connected layer state comparison unit 1541 determines that the difference is smaller than the threshold value, and if the second fully connected layer state comparison unit 1542 determines that the difference is smaller than the threshold value, the abnormal state determination unit 1500C executes an omission processing command (step ST3519).
In the omission processing command, the abnormal state judgment processing control unit 1581 instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs the second output storage unit 1590 to output the judgment value stored therein.

Abnormal state determination section 1500C executes a stored data output process (step ST3713).
The second output storage section 1590 outputs the decision value.

When the second neural network unit 1520 or the second output storage unit 1590 outputs the judgment value, the series of processes shown in FIG. 19 ends.

Next, another detailed example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment will be described.
FIG. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.

When the feature extraction section 1300C starts the process, first, the first state comparison section 1340C of the feature extraction section 1300C executes an image storage process (step ST3210).
In the image storage process, the image state comparison section 1341 of the first state comparison section 1340C stores the image that is the input data for the first neural network section 1320.
Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.

The image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST3211).
The image state comparison section 1341 refers to the storage section 1341a to determine whether the previously input image is stored therein.

When it is determined that the image has been stored previously ("YES" in step ST3211), the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST3212).
The comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.

The comparison processing unit 1341b executes a process to determine whether the difference is less than a threshold value (step ST3213). The comparison processing unit 1341b determines the magnitude of the difference between the reference value and the current value using the difference value between the currently input image and the previously input image and a pre-stored threshold value.

If the image state comparison unit 1341 determines that the value has not been stored previously (step ST3211 "NO"), or if the image state comparison unit 1341 determines that the difference between the previous value and the current value is equal to or greater than a threshold value (step ST3213 "NO"), the feature extraction processing control unit 1381C issues a normal processing command (step ST3231), and processing is performed by the convolution layer unit 1322.
Next, the convolution layer state comparison unit 1342 executes a convolution layer state storage process (step ST3214). In the convolution layer state storage process, the convolution layer state comparison unit 1342 stores input data for the convolution layer in the storage unit 1342a.

The convolutional layer state comparison unit 1342 executes a process to determine whether a previous value has been stored (step ST3215). The convolutional layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value (reference value), which is a value input previously, has been stored.

If the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that the previous state has been stored (step ST3215 "YES"), it executes a comparison process between the current state and the previous state (step ST3216).

The feature extraction unit 1300C executes a process to determine whether the difference is less than a threshold value (step ST3217). Specifically, the comparison processing unit 1342b of the convolution layer state comparison unit 1342 determines the magnitude of the difference between the previous value and the current value using the difference between the value to be input this time (current value) and the value input last time (previous value) to the pooling layer unit 1325 and a pre-stored threshold value.

If the convolutional layer state comparison unit 1342 determines that the value was not stored previously ("NO" in step ST3211), or if the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that the difference between the previous value and the current value is equal to or greater than a threshold value ("NO" in step ST3217), the feature extraction processing control unit 1381C issues a normal processing command (step ST3232), and processing is performed by the pooling layer unit 1325.
Next, in the pooling layer state storage process, the pooling layer state comparison unit 1343 stores the input data for the pooling layer (the input value to be processed) in the storage unit 1343a.

Then, the comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process to determine whether the data was previously stored (step ST3219).

If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the value was stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).

The comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process to determine whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).

If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the reference value (previous value) and the current value is equal to or greater than the threshold value (step ST3221 "NO"), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command processing, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320. The image combination unit 1328 in the first neural network unit 1320 executes image combination processing.

The feature extraction section 1300C executes an output storage process (step ST3412).
The combined image storage section 1391 of the output storage section in the feature extraction section 1300C stores the combined image output from the first neural network section 1320.

If any of the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolution layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the difference between the previous value and the current value is smaller than the threshold value (the difference value is less than the threshold value) (step ST3213 "YES", step ST3217 "YES", step ST3221 "YES"), the feature extraction unit 1300C executes an omission processing command processing (step ST3312).
In the omission processing command processing, the feature extraction processing control unit 1381 commands the first neural network unit 1320 not to execute processing in layers subsequent to the layer to be judged, and also commands the first output storage unit 1390 to output the feature map stored therein.

The feature extraction section 1300C executes a stored data output process (step ST3413).
The feature extraction unit 1300C ends the process shown in FIG.

FIG. 21 is a flowchart showing a second detailed example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
The abnormal condition determination unit 1500C starts processing when it acquires the feature map output from the feature extraction unit 1300C. The abnormal condition determination unit 1500C first executes processing on the data output by the feature extraction unit 1300C (step ST3510).

The abnormal state determination unit 1500C acquires the fully connected layer output (step ST3511).
The second state comparison section 1540 in the abnormal state determination section 1500C obtains the output value output from the first fully connected layer 1527.

The first layer state comparison unit 1541 executes a first layer state storage process (step ST3512). The first fully connected layer state comparison unit 1541 stores the output value from the first fully connected layer 1527 and the input value to the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528) as a reference value.

The first layer state comparing unit 1541 executes a process of determining whether or not the data has been previously stored (step ST3513).
The first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) is stored.

If the first layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST3513), it executes a comparison process between the current value and the previous value (step ST3514). The first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).

The first layer state comparison unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515). The comparison processing unit 1541b in the first layer state comparison unit 1541 determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.

If the first layer state comparison unit 1541 determines that no previous value was stored (step ST3211 "NO"), or if the comparison processing unit 1541b in the first layer state comparison unit 1541 determines that the difference between the current value and the reference value (previous value) is equal to or greater than the threshold value (step ST3515 "NO"), the abnormal state determination processing control unit 1581C issues a normal processing command (step ST3520), and processing is performed by the second fully connected layer 1528.

Then, the second fully connected layer state comparison unit 1542 executes a second fully connected layer state storage process (step ST3516). The second fully connected layer state comparison unit 1542 stores the output value from the second fully connected layer 1528, which is the input value to the probability output layer 1529 (the value before being processed by the probability output layer 1529), as a reference value.

Then, the second fully connected layer state comparison unit 1542 executes a process to determine whether a previous value has been stored (step ST3517). The second fully connected layer state comparison unit 1542 refers to the storage unit 1542a to determine whether a reference value (previous value) has been stored.

If the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518). The second fully connected layer state comparison unit 1542 compares the current value with a reference value (previous value).

The second fully connected layer state comparison unit 1542 executes a determination process of whether the difference is less than the threshold value (step ST3519). The comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.

If the second fully connected layer state comparison unit 1542 determines that the data was not previously stored (step ST3517 "NO"), or if the comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines that the difference is equal to or greater than the threshold value (step ST3519 "NO"), the abnormal state determination unit 1500C executes a normal processing command process (step ST3611).

Abnormal state determination section 1500C executes output processing (step ST3711).
The second neural network unit 1520 in the abnormal state determination unit 1500C outputs a determination value.

Abnormal state determination section 1500C executes an output storage process (step ST3712).
The second output storage unit 1590 in the abnormal state determination unit 1500C stores the determination value output by the second neural network unit 1520 in a probability storage unit 1591 .

If the first layer state comparison unit 1541 determines that the difference is smaller than the threshold value (less than the threshold value), and if the comparison processing unit 1542b in the second layer state comparison unit 1542 determines that the difference is smaller than the threshold value, the abnormal state determination unit 1500C executes an omission processing command (step ST3519).
In the omission processing command, the abnormal state judgment processing control unit 1581C instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.

The abnormal state determination unit 1500C executes the stored data output process (step ST3713). The second output storage unit 1590 receives a command from the abnormal state determination process control unit 1581C and outputs the determination value (probability value) stored in the probability storage unit 1591.

When the second neural network unit 1520 or the second output storage unit 1590 outputs the judgment value (probability value), the series of processes shown in FIG. 21 ends.

As described above, according to the third embodiment, if each state storage and comparison processing unit determines that the change is small, it is possible to avoid subsequent processing, thereby reducing the amount of calculations and enabling faster processing.
At this time, the result that is output is the previous highly accurate determination result, so that the output of highly accurate results can be continued.

The abnormality determination device of the present disclosure is further configured as follows.

"The neural network unit includes a first neural network unit and a second neural network unit,
The output storage unit includes a first output storage unit and a second output storage unit,
the state comparison unit includes a first state comparison unit and a second state comparison unit,
The processing control unit includes a first processing control unit and a second processing control unit,
the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
the first output storage unit that stores the feature map output by the first neural network unit;
the first state comparison unit, which stores in advance a reference value that is a reference for an input value to each of a plurality of layers in the first neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the layer;
and,
the first processing control unit instructs the first neural network unit not to execute processing in the layer of the object to be determined and subsequent layers when the first state comparison unit determines that the state of the object to be determined has not changed, and instructs the first processing control unit to output the feature map stored in the first output storage unit;
A feature extraction unit including:
the second neural network unit having a plurality of layers in the neural network, acquiring the feature map output by the feature extraction unit, and outputting a judgment value indicating a state of the object to be judged as the output data using the feature map;
the second output storage unit that stores the judgment value output by the second neural network unit;
the second state comparison unit, which stores in advance a reference value that is a reference for an input value to each layer of the second neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the layer;
and,
the second processing control unit instructing the second neural network unit not to execute processing in the layer of the object to be judged and subsequent layers when the second state comparison unit judges that the state of the object to be judged has not changed, and instructing the second neural network unit to output the judgment value stored in the second output storage unit as the output data;
An abnormality state determination unit including:
An abnormality determination device equipped with the above.

As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results.
Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.

Here, a hardware configuration for realizing the functions of the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure described above will be described.
FIG. 22 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the present disclosure.
FIG. 23 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the present disclosure.
The abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure are each realized by hardware such as that shown in FIG. 22 or FIG. 23.

As shown in FIG. 22, each of the abnormality warning devices 100 (100A, 100B, 100C) and the abnormality determination devices 1000 (1000A, 1000B, 1000C) is configured with, for example, a processor 10001, a memory 10002, and a communication circuit 10004.
The processor 10001 and the memory 10002 are installed in, for example, a computer.
The memory 10002 stores the computer, an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, image acquisition units 1100, 1100B, and 1100C, feature extraction units 1300, 1300B, and 1300C, a first neural network unit 1320, an image branching unit 1321, and a convolution layer unit 1322. , first convolutional layer 1323, second convolutional layer 1324, pooling layer section 1325, first pooling layer 1326, second pooling layer 1327, image combination section 1328, first state comparison section 1340, 1340B, 1340C, image state comparison section 1341, comparison processing section 1341b, convolutional layer state comparison section 1342, comparison processing section 1342b, pooling layer state comparison section 1343, a comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state judgment units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state judgment processing control units 1581, 1581B, 1581C, alarm output unit 2000, and programs for functioning as a control unit not shown are stored. The processor 10001 reads out and executes the programs stored in the memory 10002, thereby controlling the image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, and 1100C, the feature extraction units 1300, 1300B, and 1300C, the first neural network unit 13 20, image branching unit 1321, convolution layer unit 1322, first convolution layer 1323, second convolution layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison unit 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolution layer state comparison unit 1342, comparison Processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528 , probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and the functions of a control unit not shown are realized.
Furthermore, a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). Furthermore, the storage units 1341a, 1342a, 1343a, the first output storage unit 1390, the combined image storage unit 1391, the storage units 1541a, 1542a, the second output storage unit 1590, and the probability storage unit 1591 in the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) are realized by the memory 10002 or another memory (not shown).
Furthermore, the communication circuit 10004 realizes a communication unit (not shown).
The processor 10001 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, or a digital signal processor (DSP).
The memory 10002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk.
The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state in which data can be transmitted between them. The processor 10001, the memory 10002, and the communication circuit 10004 are also connected via the input/output interface 10003 in a state in which data can be transmitted between them and other hardware.

Or, the image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, 1100C, the feature extraction units 1300, 1300B, 1300C, the first neural network unit 1320, the image branching unit 1321, the convolution layer unit 1322, the first convolution layer 1323, the second A convolution layer 1324, a pooling layer unit 1325, a first pooling layer 1326, a second pooling layer 1327, an image combination unit 1328, a first state comparison unit 1340, 1340B, 1340C, an image state comparison unit 1341, a comparison processing unit 1341b, a convolution layer state comparison unit 1342, a comparison processing unit 1342b, a pooling layer state comparison unit 1343, a comparison processing unit 1343b, a first The functions of the processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a control unit not shown in the figure may be realized by a dedicated processing circuit 20001, as shown in FIG. 23.
Furthermore, the communication circuit 10004 realizes a communication unit (not shown).
The processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration).
Furthermore, the memory 20002 or another memory (not shown) implements a storage unit (not shown).
The memory 20002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk.
Furthermore, the communication circuit 20004 realizes a communication unit (not shown).
The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state in which they can transmit data to each other. In addition, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state in which they can transmit data to other hardware via the input/output interface 20003.
The image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, and 1100C, the feature extraction units 1300, 1300B, and 1300C, the first neural network unit 1320, the image branching unit 1321, the convolution layer unit 1322, the first convolution layer 1323, the second A convolutional layer 1324, a pooling layer unit 1325, a first pooling layer 1326, a second pooling layer 1327, an image combination unit 1328, a first state comparison unit 1340, 1340B, 1340C, an image state comparison unit 1341, a comparison processing unit 1341b, a convolutional layer state comparison unit 1342, a comparison processing unit 1342b, a pooling layer state comparison unit 1343, a comparison processing unit 1343b, The functions of the first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer state comparison unit 1542, comparison processing unit 1542b, second processing control units 1580, 1580B, 1580C, abnormal state determination processing control units 1581, 1581B, 1581C, alarm output unit 2000, and a control unit not shown in the figure may be realized by separate processing circuits, or may be realized together by a processing circuit.

Or, an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, an image acquisition unit 1100, 1100B, 1100C, a feature extraction unit 1300, 1300B, 1300C, a first neural network unit 1320, an image branching unit 1321, a convolution layer unit 1322, a first convolution layer 1323, a second convolution layer 13 24, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison unit 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolution layer state comparison unit 1342, comparison processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control unit 1380, 1380 B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C, second neural network unit 1520, state classification unit 1525, first fully connected layer 1527, second fully connected layer 1528, probability output layer 1529, second state comparison unit 1540, first fully connected layer state comparison unit 1541, comparison processing unit 1541b, second fully connected layer Some of the functions of the state comparison unit 1542, the comparison processing unit 1542b, the second processing control units 1580, 1580B, 1580C, the abnormal state determination processing control units 1581, 1581B, 1581C, the alarm output unit 2000, and a control unit (not shown) may be realized by the processor 10001 and memory 10002, and the remaining functions may be realized by the processing circuit 20001.

Note that this disclosure allows for free combinations of each embodiment, modification of any of the components of each embodiment, or omission of any of the components of each embodiment.

The present disclosure provides an abnormality determination technology that can quickly output highly accurate determination results using a neural network without performing calculations in all layers of the neural network. For example, this enables rapid and highly accurate abnormality determination, such as drowsiness detection, and is therefore suitable for application to in-vehicle driver monitoring systems.

100, 100A, 100B, 100C: abnormality warning device, 1000, 1000A, 1000B, 1000C: abnormality determination device, 1010: image acquisition unit, 1020: neural network unit, 1030: state comparison unit, 1040: processing control unit, 1050: output storage unit, 1100, 1100B, 1100C: image acquisition unit, 1300, 1300B, 1300C: feature extraction unit, 1320: neural network unit (first neural network unit), 1321: image branching unit, 1322: convolution layer unit, 1323: first convolution layer, 1324: second convolution layer, 1325 pooling layer unit 1326 first pooling layer, 1327 second pooling layer, 1328 image combination unit, 1340, 1340B, 1340C state comparison unit (first state comparison unit), 1341 image state comparison unit, 1341a storage unit, 1341b comparison processing unit, 1342 convolution layer state comparison unit, 1342a storage unit, 1342b comparison processing unit, 1343 pooling layer state comparison unit, 1343a storage unit, 1343b comparison processing unit, 1380, 1380B, 1380C processing control unit (first processing control unit), 1381, 13 81B, 1381C: feature extraction processing control section, 1390: output storage section (first output storage section), 1391: combined image storage section, 1500, 1500B, 1500C: abnormal state determination section, 1520: neural network section (second neural network section), 1525: state classification section, 1527: first fully connected layer, 1528: second fully connected layer, 1529: probability output layer, 1540: state comparison section (second state comparison section), 1541: first layer state comparison section (first fully connected layer state comparison section), 1541a: storage section, 1541b: comparison processing section, 1542: second layer state Comparison unit (second fully connected layer state comparison unit), 1542a storage unit, 1542b comparison processing unit, 1580, 1580B, 1580C processing control unit (second processing control unit), 1581, 1581B, 1581C abnormal state determination processing control unit, 1590 output storage unit (second output storage unit), 1591 probability storage unit, 2000 alarm output unit, 10001 processor, 10002 memory, 10003 input/output interface, 10004 communication circuit, 20001 processing circuit, 20002 memory, 20003 input/output interface, 20004 communication circuit.

Claims (14)

an image acquisition unit that acquires and outputs an image;
a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image;
an output storage unit that stores output data output by the neural network unit;
a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
a processing control unit that instructs the neural network unit not to execute processing in the first layer and subsequent layers when the state comparison unit determines that the state of the object to be determined has not changed, and instructs the neural network unit to output the output data stored in the output storage unit;
An abnormality determination device comprising: the neural network unit includes a first neural network unit and a second neural network unit;
The output storage unit includes a first output storage unit and a second output storage unit,
the state comparison unit includes a first state comparison unit and a second state comparison unit,
The processing control unit includes a first processing control unit and a second processing control unit,
the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
the first output storage unit that stores the feature map output by the first neural network unit;
the first state comparison unit pre-stores a reference value that is a reference for an input value to a second layer that is at least one of the layers in the first neural network unit, and judges a change in the state of the object to be judged by using a difference value between the reference value and a current value that is a value that is to be newly input to the second layer;
and,
the first processing control unit instructs the first neural network unit not to execute processing in the second layer and subsequent layers of the object to be determined when the first state comparison unit determines that the state of the object to be determined has not changed, and instructs the first processing control unit to output the feature map stored in the first output storage unit;
A feature extraction unit including:
the second neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring the feature map output by the feature extraction unit, and outputting a judgment value indicating a state of the object to be judged as the output data using the feature map;
the second output storage unit that stores the judgment value output by the second neural network unit;
the second state comparison unit stores in advance a reference value that is a reference for an input value to a third layer that is at least one of the layers in the second neural network unit, and judges a change in the state of the object to be judged by using a difference value between the reference value and a current value that is a value that is to be newly input to the third layer;
and,
the second processing control unit instructs the second neural network unit not to execute processing in the third layer and subsequent layers of the object to be judged when the second state comparison unit judges that the state of the object to be judged has not changed, and instructs the second processing control unit to output the judgment value stored in the second output storage unit as the output data;
An abnormality state determination unit including:
The abnormality determination device according to claim 1 . the neural network unit includes a first neural network unit and a second neural network unit;
The output storage unit includes a first output storage unit and a second output storage unit,
the state comparison unit includes a first state comparison unit and a second state comparison unit,
The processing control unit includes a first processing control unit and a second processing control unit,
the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
the first output storage unit that stores the feature map output by the first neural network unit;
the first state comparison unit, which stores in advance a reference value that is a reference for an input value to each of a plurality of layers in the first neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the layer;
and,
the first processing control unit instructs the first neural network unit not to execute processing in the layer of the object to be determined and subsequent layers when the first state comparison unit determines that the state of the object to be determined has not changed, and instructs the first processing control unit to output the feature map stored in the first output storage unit;
A feature extraction unit including:
the second neural network unit having a plurality of layers in the neural network, acquiring the feature map output by the feature extraction unit, and outputting a judgment value indicating a state of the object to be judged as the output data using the feature map;
the second output storage unit for storing the judgment value output by the second neural network unit;
the second state comparison unit, which stores in advance a reference value that is a reference for an input value to each layer of the second neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the layer;
and,
the second processing control unit instructing the second neural network unit not to execute processing in the layer of the object to be judged and subsequent layers when the second state comparison unit judges that the state of the object to be judged has not changed, and instructing the second neural network unit to output the judgment value stored in the second output storage unit as the output data;
An abnormality state determination unit configured to include
The abnormality determination device according to claim 1 , The abnormality determination device according to any one of claims 1 to 3, characterized in that the reference value is a previous value stored for each layer as a result of a previous process in the neural network. The reference value is a typical output value of a pre-modeled intermediate layer in the neural network.
4. The abnormality determination device according to claim 1, wherein the abnormality determination device is a detection device for detecting an abnormality. The reference value is an output value of a predetermined node among the nodes included in the neural network.
4. The abnormality determination device according to claim 1, wherein the abnormality determination device is a detection device for detecting an abnormality. The state comparison unit is
a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value;
2. The abnormality determination device according to claim 1 . The state comparison unit is
When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
8. The abnormality determination device according to claim 7. At least one of the first state comparison unit and the second state comparison unit is
a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value;
4. The abnormality determination device according to claim 2 or 3. At least one of the first state comparison unit and the second state comparison unit is
When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
10. The abnormality determination device according to claim 9. The state comparison unit is
When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
8. The abnormality determination device according to claim 7. At least one of the first state comparison unit and the second state comparison unit is
When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
10. The abnormality determination device according to claim 9. The device further includes an alarm output unit that acquires the output data, which is a judgment value indicating a drowsy state or a dozing state of the person to be judged, and outputs an alarm to the person to be judged according to the judgment value.
The abnormality determination device according to any one of claims 1 to 12. The image acquisition unit outputs the acquired image to the neural network unit,
the neural network unit having a plurality of layers in a neural network acquires the image output by the image acquisition unit, and outputs a judgment result indicating a state of a judgment target included in the image;
an output storage unit that stores the output data output by the neural network unit;
a state comparison unit pre-stores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
When the state comparison unit determines that the state of the object to be determined has not changed, the processing control unit instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs the neural network unit to output the output data stored in the output storage unit.
Method for determining abnormality.

Images