WO2025099876A1 - Information processing device, information processing method, and information processing program - Google Patents

Abstract

This information processing device makes inferences (e.g., object detection) by decimating consecutive frames on a prescribed interval. Upon detecting an event by inference with respect to a frame, the information processing device makes inferences with respect to consecutive frames to the frame in which the event was detected. If the event is no longer detected as a result of the inferences with respect to the frames, the information processing device again makes inferences by decimating frames on the prescribed interval. The information processing device also makes inferences by going back to a frame preceding the frame in which the event was detected.

Description

Information processing device, information processing method, and information processing program

The present invention relates to an information processing device, an information processing method, and an information processing program for performing inference on video frames.

　With the development of deep learning and the spread of IoT (Internet of Things) devices, applications have emerged that detect people from videos captured by surveillance cameras installed throughout the city and detect vehicles from videos captured by in-vehicle cameras.

Here, to ensure that the above video processing can be done in real time, it is desirable to perform the processing on the IoT device. However, IoT devices have significant resource constraints, making it difficult to deploy deep learning models for highly accurate object detection, etc.

In response, a technology has been proposed in which IoT devices and a cloud computer work together to perform object detection processing. However, this technology has the problem that as the number of IoT devices increases, the inference costs of the entire system increase accordingly. To solve this problem, a technology has been proposed in which, for example, the IoT device (first layer) first performs inference processing such as object detection for videos using a lightweight model, and the cloud computer (second layer) performs advanced inference processing only when it is determined to be necessary based on the output results of the above inference processing (two-layer inference system, see non-patent document 1).

Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, Matei Zaharia, "NoScope: Optimizing Neural Network Queries over Video at Scale", [online], Internet <URL: https://www.vldb.org/pvldb/vol10/p1586-kang.pdf>

However, in the two-layer inference system described above, if the inference model used in the first layer is made lightweight in order to reduce the resources of the IoT device (first layer), the inference accuracy of the first layer will decrease, which will in turn affect the inference accuracy of the entire system.

For example, if the input data actually contains the object to be detected, but the first layer infers that "there is a low probability that the input data contains the object to be detected," then inference will not be performed on that data in the second layer. As a result, the object to be detected that is contained in the input data will go undetected.

The present invention aims to reduce the frequency of inference without reducing the accuracy of inference when performing inference processing on data such as video data, which is likely to have consecutive events.

In order to solve the above problems, the present invention is characterized by comprising a data input unit that accepts input of consecutive data, an inference unit that thins out the consecutive data and performs inference on the data, and performs inference on consecutive data starting from data at which an event is detected by the inference, and an output processing unit that outputs the inference results based on the inference on the data.

According to the present invention, when performing inference processing on data such as video data in which there is a high possibility of consecutive events occurring, it is possible to reduce the frequency of inference without reducing the accuracy of the inference.

FIG. 1 is a diagram for explaining an overview of an information processing device. FIG. 2 is a diagram for explaining an example of frame inference. FIG. 3 is a diagram illustrating an example of a process executed by the information processing device. FIG. 4 is a diagram illustrating an example of the configuration of an information processing device. FIG. 5 is a flowchart illustrating an example of a process executed by the information processing device. FIG. 6 is a diagram illustrating an example of a process in which the information processing device performs inference by going back to past frames. FIG. 7 is a diagram illustrating an example of a process in which an information processing device executes inference on each frame of a video file. FIG. 8 is a diagram illustrating an example of the configuration of a computer that executes an information processing program.

Below, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment.

First, an overview of the information processing device of this embodiment will be described with reference to FIG. 1. Note that the data to be processed by the information processing device is, for example, continuous data output from an IoT device. In the following, an example will be described in which the data is a video (continuous frames) captured by a surveillance camera or the like.

Among the frames that make up a video, a characteristic is that an event that occurs in a certain frame is highly likely to have also occurred in adjacent frames. The inventors focused on this characteristic and configured an information processing device to perform inference for each frame in the following manner.

The information processing device first performs inference while thinning out the input data (consecutive frames). The inference here is, for example, the detection of an object reflected in the frame (see Figure 2). Then, if the information processing device determines that a specific event has occurred as a result of the frame inference (if it determines that an event has occurred), it performs frame inference continuously from the next frame onwards.

For example, if the information processing device determines that an object with a predetermined attribute (class) is captured in a predetermined area of frame 201 based on the inference result (reference symbol 202) for frame 201 shown in FIG. 2, it determines that an event has occurred, and executes frame inference continuously from the next frame onwards. On the other hand, if the information processing device determines that a predetermined event has not occurred as a result of frame inference (determines that there is no event), it thins out frames and executes inference in the same way as before.

In other words, the information processing device determines whether to thin out inferences for a frame based on whether a specified event has occurred in the frame. This enables the information processing device to reduce the frequency of inferences for a frame and prevent missed detections.

A specific example of processing executed by an information processing device will be described with reference to FIG. 3. As shown in FIG. 3, the information processing device performs inference by thinning out consecutive frames (for example, performing inference every 5 frames), and if the inference result shows that no event is detected from the frame, the inference is skipped until the next inference timing. On the other hand, if the inference result shows that an event is detected from the frame, the information processing device performs inference on consecutive frames. Then, if the inference result shows that no event is detected, the information processing device performs inference by thinning out frames again.

The information processing device may infer frames consecutively, and when an event is no longer detected in even one frame, it may resume the process of thinning out frames and inferring, or when an event is no longer detected in multiple consecutive frames (e.g., two frames), it may resume the process of thinning out frames and inferring.

[Configuration example]
Next, a configuration example of the information processing device 10 will be described with reference to Fig. 4. The information processing device 10 includes, for example, an input/output unit 11, a communication unit 12, a storage unit 13, and a control unit 14.

The input/output unit 11 is an interface that handles the input and output of various data. For example, the input/output unit 11 accepts input of various settings to the information processing device 10.

The communication unit 12 is a communication interface for inputting and outputting various types of data via a network. For example, the communication unit 12 receives data (successive frames, etc.) output from an IoT device via a network.

The memory unit 13 stores data, programs, etc. that are referenced when the control unit 14 executes various processes. The memory unit 13 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

The storage unit 13 stores various information, such as parameters of a deep learning model used by the inference unit 142 (described below) when inferring data. For example, the storage unit 13 stores setting information such as how many frames to thin out when the inference unit 142 thins out frames to perform inference (thinning amount), how many consecutive frames of no event detection are required before thinning out frames again to perform inference (number of undetected frames), etc.

The storage unit 13 also stores definition information for events to be detected. The event definition information is information that defines, for example, the number of detected target objects in the frame, the detection size, and the degree of certainty required to determine that an event has occurred. The above setting information, definition information for events to be detected, etc. can be changed as appropriate by the user of the information processing device 10, etc.

The control unit 14 is responsible for controlling the entire information processing device 10. The functions of the control unit 14 are realized, for example, by the CPU (Central Processing Unit) executing a program stored in the storage unit 13.

The control unit 14 includes, for example, a data input unit 141, an inference unit 142, and an output processing unit 143. The data input unit 141 accepts input of continuous data. For example, the data input unit 141 accepts input of data (continuous frames) captured by a surveillance camera via the communication unit 12.

The inference unit 142 performs inference on the consecutive data received by the data input unit 141 by thinning it out. Then, when the inference unit 142 detects an event by inference on the data, it performs inference on the consecutive data starting from the data where the event was detected.

For example, the inference unit 142 performs inference by thinning out consecutive frames received by the data input unit 141 at a predetermined interval. Then, if the inference unit 142 detects an event indicated in the event definition information as a result of the inference for a frame, it performs inference continuously for the subsequent frames. Then, if the inference unit 142 determines as a result of the inference for a frame that it has no longer detected an event, it stops inference for the consecutive frames and performs inference again by thinning out the frames.

For example, when the inference unit 142 uses the inference result of the previous frame to make an event judgment (event present/no event) for the previous frame, it assigns the judgment result to the next frame as metadata (inference necessity flag). Then, if the inference necessity flag of the frame is "necessary", the inference unit 142 performs inference for that frame, and if the inference necessity flag is "no", it does not perform inference for that frame.

The frame inference performed by the inference unit 142 includes, for example, identifying the BBOX (position information) of an object captured in the frame, identifying the class of the object in the BBOX (what the object is), and calculating the confidence level of the BBOX. For frame inference, for example, a deep learning model for object detection from an image is used.

The output processing unit 143 outputs the inference results of the frames by the inference unit 142. For example, the output processing unit 143 outputs each piece of data (frame) output from the inference unit 142 by adding meta-information indicating the inference results.

[Example of processing procedure]
Next, an example of a processing procedure executed by the information processing device 10 will be described with reference to FIG. 5. For example, when the data input unit 141 of the information processing device 10 receives input of continuous data (S1), the inference unit 142 performs inference while thinning out the data (S2). Then, when the inference unit 142 determines that an event has occurred based on the result of the data inference (Yes in S3), it performs inference continuously from the next data onwards (S4). Then, it returns to S3. On the other hand, when the inference unit 142 determines that no event has occurred based on the result of the data inference (No in S3), it returns to S2 and performs inference while thinning out the data from the next data onwards (S2).

By executing the above process, the information processing device 10 can reduce the frequency of inference for the data to be inferred (continuous data) and prevent missed detections.

If the information processing device 10 determines that an event has occurred as a result of frame inference, there is a possibility that the event has also occurred in a past frame for which inference was skipped. Therefore, when the information processing device 10 detects an event by inferring data, it may perform inference retroactively on the skipped past data (frames).

In this case, the inference unit 142 performs inference by going back to past frames starting from the frame in which the event is detected, as shown in FIG. 6. For example, the inference unit 142 performs inference on past frames for which inference has not been performed, starting from the frame immediately before the frame in which the event is detected.

In this way, the information processing device 10 can detect events without omission for the frames being processed.

When going back in time to perform frame inference as described above, the inference unit 142 may perform batch processing at once on multiple past frames (e.g., four frames) for which inference has not yet been performed. In this way, the inference unit 142 can efficiently perform inference on past frames.

The inference unit 142 may perform 1. inference from the frame in which the event was detected to frames beyond (inference for frames in the future direction) and 2. inference from the frame in which the event was detected to frames past (inference for frames in the past direction) in the order of 1 → 2 or 2 → 1.

[Example of application to video files]
The information processing device 10 may perform the above processing on frames sequentially output from an IoT device, or may perform the above processing on a set of frames for a predetermined period of time (e.g., a video file). An example of the case where the information processing device 10 performs processing on a video file will be described with reference to FIG. 7.

For example, when the information processing device 10 executes processing on a video file, it is possible to check from any frame at any time within the video. Thus, for example, the inference unit 142 performs regular checks of future frames within the video file (detecting an event by thinning out frames at a predetermined interval from the first frame and inferring) as shown in FIG. 7 ((1)), and when an event is detected, performs event checks continuously from the frame where the event was detected to future frames ((2)).

After that, when the inference unit 142 has finished checking up to the first frame in the future, it then performs event checks continuously on the frames in the past ((3)).

In this way, the inference unit 142 can, for example, detect events in a long video file starting from the most recent events.

In addition, in the above (2) and (3), the inference unit 142 may determine non-detection of an event, for example, using the results of inference for multiple frames ((4) Non-detection determination based on results for multiple frames). For example, if the inference unit 142 does not detect an event in two consecutive frames, the event determination may be "no" and consecutive event checks may be stopped.

The amount of frames that the inference unit 142 will thin out before making an inference (thinning amount) and the amount of consecutive frames over which an event is no longer detected before making an inference (number of undetected frames) may be fixed values or may be values that change dynamically.

For example, if the input data is video captured by an on-board camera, the inference unit 142 sequentially acquires information on the speed of the passenger vehicle in which the on-board camera is mounted. Then, if the speed of the passenger vehicle increases, the inference unit 142 may reduce the amount of thinning in accordance with the speed.

By doing this, even if the speed of the passenger vehicle equipped with the onboard camera increases and the changes in each frame of the video become greater than before, the inference unit 142 infers frames at shorter intervals accordingly, thereby reducing missed detections.

Furthermore, when the speed of the passenger vehicle decreases, the inference unit 142 increases the amount of frame thinning in accordance with the speed. In this way, when the change in each frame of the video becomes smaller than before due to a decrease in the speed of the passenger vehicle equipped with an on-board camera, the inference unit 142 can perform frame inference less frequently. As a result, the inference unit 142 can perform frame inference by making effective use of limited computational resources.

In addition, the amount of thinning and the number of non-detection frames may be dynamically changed based on the confidence level of the BBOX included in the inference result of the frame.

For example, if the result of frame inference by the inference unit 142 is that the certainty of the BBOX is just below the event determination threshold (for example, the frame appears to show something that looks like a person, but the certainty is not very high), the amount of frame thinning may be reduced or the number of undetected frames may be increased.

According to the information processing device 10 described above, for example, there is no degradation in frame inference accuracy due to the low-precision model in the first layer that occurs in the two-layer inference system described above. Furthermore, the information processing device 10 thins out frames and performs inference when an event has not yet occurred, thereby reducing the frequency of inference.

In addition, the information processing device 10 does not require a model equivalent to the first layer model in a two-layer inference system. Therefore, in the case of the information processing device 10, there is no need to implement or tune a model equivalent to the first layer model in a two-layer inference system. Furthermore, if events occur frequently in each frame, the two-layer inference system uses first layer resources + second layer resources, but the information processing device 10 only needs resources equivalent to the second layer resources. Therefore, with the information processing device 10, it is possible to reduce the resources required for frame inference compared to conventional methods.

[System configuration, etc.]
In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
The information processing device 10 can be implemented by installing a program (information processing program) as package software or online software on a desired computer. For example, the information processing device can function as the information processing device 10 by executing the above program on the information processing device. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and further terminals such as PDAs (Personal Digital Assistants).

FIG. 8 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process executed by the information processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the information processing device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

The data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes it.

The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 10 Information processing device 11 Input/output unit 12 Communication unit 13 Storage unit 14 Control unit 141 Data input unit 142 Inference unit 143 Output processing unit

Claims (7)

a data input unit for accepting input of continuous data;
an inference unit that thins out the continuous data and performs inference on the data, and performs inference on continuous data starting from data in which an event is detected by the inference;
and an output processing unit that outputs an inference result based on the inference of the data. The inference unit is
The information processing apparatus according to claim 1 , further comprising: a processor configured to execute inference on the continuous data until the event is no longer detected. The inference unit is
The information processing device according to claim 1, characterized in that, starting from the data at which an event is detected by the inference, inference is performed on consecutive data from the data and on data older than the data at which the inference has not been performed. The inference unit is
The information processing apparatus according to claim 3 , further comprising: performing inference by batch processing on a plurality of data older than the data and not yet subjected to the inference. The information processing apparatus according to claim 1 , wherein the data is frames constituting a moving image. An information processing method executed by an information processing device,
accepting input of continuous data;
a step of thinning out the continuous data and performing inference on the data, and performing inference on continuous data starting from data in which an event is detected by the inference;
and outputting an inference result based on the inference of the data. accepting input of continuous data;
a step of thinning out the continuous data and performing inference on the data, and performing inference on continuous data starting from data in which an event is detected by the inference;
and outputting an inference result based on the inference of the data.

Images