JP7667723B2 - Information processing device and information processing method - Google Patents

Description

This disclosure relates to an information processing device and an information processing method.

Conventionally, as shown in, for example, Japanese Patent Application Laid-Open No. 7-146897 (Patent Document 1), a system is known that classifies the tasks performed in a workplace while playing back video of the workplace captured by an infrared video camera.

In more detail, in this system, the contents of a video tape recorded by an infrared video camera are played back on a video deck. The human body recognition unit in the system's controller tracks the red area displayed on the infrared video monitor that corresponds to the worker, and inputs the movement of the red area into the condition comparison unit via the mode switching unit. The condition comparison unit in the controller reads the conditions stored in the condition memory unit, compares them with the data sent from the human body recognition unit, and classifies the work being performed by the worker.

Japanese Unexamined Patent Publication No. 7-146897

The system in Patent Document 1 requires the subject to be captured by an infrared video camera, making it less versatile.

The present disclosure provides an information processing device and information processing method that can determine the type of work being performed in a workplace based on video image data (multiple frame image data) captured by a visible light camera.

According to one aspect of the present disclosure, an information processing device includes an acquisition means for acquiring a series of multiple frame image data captured by an image capture using a visible light camera. The visible light camera is fixed in position and orientation, and captures an image of a workplace where a worker is performing work involving blinking light as a subject. The information processing device further includes a detection means for detecting an area of the worker in a first frame image data of the multiple frame image data, a generation means for generating image data showing a change in the state of the subject based on the first frame image data and a second frame image data of the multiple frame image data that is a predetermined number of frames before the first frame image data, an extraction means for extracting image data of an area corresponding to the detected area of the worker from the generated image data, and a determination means for determining the type of work being performed in the workplace based on the extracted image data.

According to another aspect of the present disclosure, the information processing method includes a step of capturing an image of a workplace where a worker is performing work involving blinking light, as a subject, by a visible light camera. The visible light camera is fixed in its installation position and attitude. The information processing method further includes a step of acquiring a plurality of consecutive frame image data obtained by capturing images by the visible light camera, a step of detecting an area of the worker in a first frame image data of the plurality of frame image data, a step of generating image data showing a change in the state of the subject based on the first frame image data and a second frame image data of the plurality of frame image data that is a predetermined number of frames before the first frame image data, a step of extracting image data of an area corresponding to the area of the detected worker from the generated image data, and a step of determining a type of work being performed in the workplace based on the extracted image data.

According to the present disclosure, it is possible to determine the type of work being performed in a workplace based on multiple frame image data obtained by capturing images using a visible light camera.

FIG. 1 is a diagram for explaining a schematic configuration of a determination system. FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. FIG. 2 is a diagram for explaining an overview of a process executed by an information processing device. FIG. 11 is a flowchart showing the flow of a determination process. FIG. 5 is a flow chart showing details of the process in step S1 of FIG. 4. FIG. 5 is a flowchart showing details of the process in step S2 of FIG. 4. FIG. 7 is a flowchart for explaining details of the process of step S202 in FIG. 6. FIG. 8 is a diagram for explaining the process of FIG. 7 using image data. FIG. 7 is a flowchart for explaining details of the process of step S206 in FIG. 6. 13 is a diagram showing data including the final judgment result stored in memory in a highly compatible file format. FIG. FIG. 8 is a flowchart showing a modified example of the series of processes in step S202 shown in FIG. 7.

The following describes the embodiment with reference to the drawings. In the following description, identical parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated.

First, we will explain some of the terms used in this embodiment.

"Trained model" refers to an "inference program" that incorporates "trained parameters." "Trained parameters" refer to parameters (coefficients) obtained as a result of learning using a training dataset. Trained parameters are generated by inputting the training dataset into a training program and mechanically adjusting it for a certain purpose. "Inference program" refers to a program that makes it possible to output a certain result for an input by applying the incorporated trained parameters.

"Learning program" refers to a program that executes an algorithm to find certain rules from a training dataset and generate a model that expresses those rules. Specifically, this refers to a program that specifies the procedures to be executed by a computer in order to realize learning using the adopted learning method.

A "learning dataset" refers to secondary processed data that is generated by converting and/or processing raw data through preprocessing such as removing missing values and outliers, adding separate data such as label information (ground truth data), or combining these to facilitate analysis using a target learning method. A learning dataset is a collection of data (hereinafter also referred to as "learning data"). In this embodiment, the learning data includes one image data and label information.

A. System Configuration
FIG. 1 is a diagram for explaining a schematic configuration of a determination system according to the present embodiment.

As shown in FIG. 1, the determination system 1000 includes a camera 1 and an information processing device 2.

Camera 1 is a visible light camera. The installation position and posture of camera 1 are fixed. Camera 1 captures an image of a workplace where a worker 900 is performing work involving blinking light as a subject. Examples of work involving blinking light include welding work, grinding work, and gouging work. Note that welding work includes spot welding.

Movie data Da captured by camera 1 is sent to information processing device 2. The movie data Da is composed of multiple consecutive frame image data #1, #2, #3, ....

The information processing device 2 is used by a user 950. The information processing device 2 is typically a personal computer. The information processing device 2 acquires moving image data Da captured by the camera 1 from the camera 1. The information processing device 2 may acquire the moving image data Da via another device such as a server device. The information processing device 2 may also acquire the moving image data Da via a storage medium such as an IC card or a USB memory.

Figure 2 shows the hardware configuration of information processing device 2.

2, the information processing device 2 includes a processor 201, a memory 202, a display 203, an input device 204, a communication interface 205, a card reader 206, and a USB port 207. The memory 202 includes a ROM (Read Only Memory) 221, a RAM (Random Access Memory) 222, a SSD (Solid State Drive) 223, and a HDD (Hard Disk Drive) 224.

The memory 202 stores an operating system and various programs including a trained model. The memory 202 stores programs for executing various processes described below. The processor 201 executes the operating system and the above programs.

The input device 204 accepts operational input from the user 950. The input device 204 is typically a keyboard or a mouse.

The processor 201 executes various processes based on operations received by the input device 204. The processor 201 displays various information on the display 203. The processor 201 displays the results of program execution on the display.

The communication interface 205 is an interface for communicating with external devices. The processor 201 acquires video image data Da from the camera 1 via the communication interface 205.

The card reader 206 reads data stored on an IC card. A USB memory is connected to the USB port 207. The processor 201 can also acquire video image data Da via the card reader 206 or the USB port 207.

<B. Determination of work type>
FIG. 3 is a diagram for explaining an overview of the process executed by the information processing device 2. As shown in FIG.

As shown in FIG. 3, the information processing device 2 includes a video data acquisition unit 10, a person area detection unit 20, a generation unit 30, an extraction unit 40, an operation type determination unit 50, a storage unit 60, a display control unit 70, and a display unit 80.

The video data acquisition unit 10 corresponds to the communication interface 205, the card reader 206, or the USB port 207. The person area detection unit 20, the generation unit 30, the extraction unit 40, the task type determination unit 50, and the display control unit 70 are functional blocks that are realized by the processor 201 executing programs and the like stored in the memory 202. The storage unit 60 corresponds to the memory 202. The display unit 80 corresponds to the display 203.

The video data acquisition unit 10 acquires video data Da obtained by imaging using the camera 1 from the camera 1. In detail, the video data acquisition unit 10 acquires a series of multiple frame image data #1, #2, #3, ... obtained by the camera 1.

In the information processing device 2, the detection process by the person area detection unit 20 and the image generation process by the generation unit 30 are performed separately for a plurality of frame image data #1, #2, #3, .... Note that the detection process by the person area detection unit 20 and the image generation process by the generation unit 30 may be performed either first or simultaneously.

In detail, the video data Da acquired by the video data acquisition unit 10 is temporarily stored in a memory 202 such as the storage unit 60, and is then read out by the person area detection unit 20 and the generation unit 30.

The following mainly describes the person area detection unit 20, the generation unit 30, the extraction unit 40, the task type determination unit 50, and the display control unit 70.

(b1. Person area detection unit 20)
The person area detection unit 20 detects the area of the worker (hereinafter also referred to as "person area") in each of the multiple frame image data #1, #2, #3, ... (hereinafter referred to as "frame image data #N" where N is a natural number). In this example, the person area detection unit 20 is realized by the trained model M20. Note that in this example, the person area in frame image data #1 is not used, so there is no need to detect the person area in frame image data #1.

The trained model M20 receives frame image data #N as input and outputs information indicating a person area. Specifically, the trained model M20 outputs the coordinates of the person area. More specifically, the trained model M20 extracts a rectangular area as the person area. Typically, the area can be expressed as the coordinate values of the four corners of the rectangle. In this way, the trained model M20 outputs four coordinate values as the coordinates of the person area.

The person area detection unit 20 sends the coordinates of the person area to the extraction unit 40 along with an identifier (e.g., frame number, timestamp) indicating frame image data #N. Specifically, the person area detection unit 20 sends the coordinates of the person area in frame image data #N associated with the identifier of frame image data #N to the extraction unit 40. For example, if the video data Da contains K frame image data (1≦N≦K), the person area detection unit 20 sends a set of coordinates of K person areas (coordinate values are 4K=4×K in total) to the extraction unit 40. More specifically, in this example, the coordinates of the K person areas are temporarily stored in memory 202 such as the storage unit 60, and then read out by the extraction unit 40.

(b2. Generation unit 30)
The generating unit 30 generates image data showing a change in the subject's state based on frame image data #N and frame image data #N-1, which is one frame before frame image data #1. More specifically, the generating unit 30 generates image data showing a change in the subject's state by a frame difference method (also called an "inter-frame difference method"). Note that since frame image data #0 does not exist, image data showing a change in the subject's state based on frame image data #0 and frame image data #1 is not generated.

More specifically, the generator 30 generates masked image data #N-1 (2≦N≦K) from consecutive frame image data #N-1 and frame image data #N using the frame difference method. The frame difference method and mask processing will be described later.

The generating unit 30 generates multiple pieces of image data #N-1 after mask processing. In this way, the generating unit 30 generates multiple pieces of image data, each of which indicates a change in the state of the subject.

The generation unit 30 sends the masked image data #N-1 to the extraction unit 40 along with an identifier indicating the frame image data #N. Specifically, the generation unit 30 sends the masked image data #N-1 associated with the identifier of the frame image data #N to the extraction unit 40. For example, if the video image data Da contains K frame image data, the generation unit 30 sends the K-1 masked image data to the extraction unit 40. More specifically, in this example, the K-1 masked image data are temporarily stored in a memory 202 such as the storage unit 60, and are then read out by the extraction unit 40.

(b3. Extraction Unit 40)
The extraction unit 40 extracts image data of an area corresponding to the person area detected by the person area detection unit 20 from the image data (masked image data #N-1) generated by the generation unit 30. In other words, the extraction unit 40 cuts out the image.

In detail, the extraction unit 40 uses the coordinates of the person area and the masked image data, which are assigned the same identifier, to extract image data of a rectangular area corresponding to the person area from the masked image data.

Specifically, the extraction unit 40 extracts image data of an area (more specifically, a rectangular area specified by the coordinates of the four vertices of the person area) corresponding to the person area detected in frame image data #2 from post-mask processing image data #1 generated by applying the frame difference method to frame image data #1 and frame image data #2. Similarly, the extraction unit 40 extracts image data of an area corresponding to the person area detected in frame image data #3 from post-mask processing image data #2 generated by applying the frame difference method to frame image data #2 and frame image data #3. If the moving image data Da contains K pieces of frame image data, the extraction unit 40 performs this type of extraction process K-1 times in total.

The extraction unit 40 sends the extracted image data #N-1 to the work type determination unit 50. In detail, in this example, the K-1 extracted image data are temporarily stored in a memory 202 such as the storage unit 60, and are then read out by the work type determination unit 50.

(b4. Work type determination unit 50)
The work type determination unit 50 determines the type of work being performed in the workplace based on the image data extracted by the extraction unit 40. In other words, the work type determination unit 50 classifies the works being performed in the workplace. The work type determination unit 50 includes a welding work determination unit 51, a grinding work determination unit 52, a gouging work determination unit 53, and a final determination unit 54.

Welding is a joining process that joins metals together. When welding is performed, light is emitted, called arc light. Grinding is a grinding process that cuts metal with a grindstone or the like. When grinding is performed, the shaved metal powder emits light. Gouging is a process that cuts, melts, and removes metal. When gouging is performed, light is emitted by arc discharge. In this way, welding, grinding, and gouging are all metal processing, and light is emitted from the part being processed as the work is performed. However, welding, grinding, and gouging each have different light emission states. In this embodiment, the information processing device 2 focuses on the difference in the light emission states and determines the work type.

The welding work determination unit 51 determines whether the work being performed in the workplace is welding work or not based on the image data #N-1 extracted by the extraction unit 40. In more detail, in this example, the welding work determination unit 51 is realized by the trained model M51.

The trained model M51 receives as input the image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being performed in the workplace is welding work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, the trained model M51 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being performed in the workplace is welding work.

The trained model M51 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs it to the final determination unit 54. For example, if the video image data Da contains K frame image data, the trained model M51 outputs K-1 accuracy values to the final determination unit 54.

The grinding operation determination unit 52 determines whether the work being performed in the workplace is grinding operation based on the image data #N-1 extracted by the extraction unit 40. In more detail, in this example, the grinding operation determination unit is realized by the trained model M52.

The trained model M52 receives as input the image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being done in the workshop is grinding work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, like the trained model M51, the trained model M52 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being done in the workshop is grinding work.

Like the trained model M51, the trained model M52 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs it to the final determination unit 54. For example, if the video image data Da contains K frame image data, the trained model M52 outputs K-1 accuracy rates to the final determination unit 54.

The gouging operation determination unit 53 determines whether the work being performed in the workplace is gouging work based on the image data #N-1 extracted by the extraction unit 40. In more detail, in this example, the gouging operation determination unit is realized by the trained model M53.

The trained model M53 receives as input the image data #N-1 (one piece of image data) extracted by the extraction unit 40, and outputs a degree of accuracy indicating that the work being performed in the workplace is gouging work. In this example, the degree of accuracy is a value between 0 and 1 inclusive. In this way, like the trained models M51 and M52, the trained model M53 normalizes the degree of accuracy (in this example, the minimum value is 0 and the maximum value is 1) and outputs it. The higher the degree of accuracy, the more likely it is that the work being performed in the workplace is gouging work.

Similar to trained models M51 and M52, trained model M53 associates the accuracy calculated for each extracted image data #N-1 with the identifier of the extracted image data #N-1 and outputs it to final determination unit 54. For example, if video image data Da contains K frame image data, trained model M53 outputs K-1 accuracy rates to final determination unit 54.

The final determination unit 54 determines whether the work being performed in the workplace is welding work, grinding work, gouging work, or work that cannot be classified, based on the results of the determination by the trained model M51, the results of the determination by the trained model M52, and the results of the determination by the trained model M53. Examples of other work include the movement of workers.

In more detail, the final determination unit 54 determines whether or not there is any work whose accuracy is equal to or greater than a threshold value (e.g., 0.6) for each extracted image data #N-1 (each piece of image data). If there is any work whose accuracy is equal to or greater than the threshold value, the final determination unit 54 determines that the work is being performed in a workplace.

For example, if the accuracy output from trained model M51 for one extracted image data #1 is 0.7, the accuracy output from trained model M52 is 0.1, and the accuracy output from trained model M53 is 0.05, the final judgment unit 54 judges that the work being performed in the workplace is welding work. Also, if the accuracy output from trained model M51 for one extracted image data #2 is 0.5, the accuracy output from trained model M52 is 0.2, and the accuracy output from trained model M53 is 0.1, there is no accuracy greater than or equal to the threshold value (0.6 in this example), so the final judgment unit 54 judges that the work being performed in the workplace cannot be classified. Such a judgment is made for each extracted image data #N-1.

Furthermore, the final judgment unit 54 performs the final judgment at a predetermined cycle (for example, every second). The cycle can be set appropriately based on the frame rate of the video data Da. For example, the frame rate of the video data Da is 60 fps (frames per second). In this case, the video data Da includes 60 frame image data per second.

Therefore, the final determination unit 54 obtains 60 determination results during one second of the video image data Da. The final determination unit 54 determines (hereinafter also referred to as the "final determination") that the work type with the greatest number of results among the 60 determination results is the work being performed in the workplace during that period (one second).

For example, if, out of 60 judgment results in a one-second period, there were 40 judgments of welding work, 4 judgments of grinding work, 0 judgments of gouging work, and 16 judgments that classification was not possible, the final judgment unit 54 will judge the work type for that period to be welding (final judgment).

The final judgment unit 54 stores the result of the final judgment in the storage unit 60. In particular, the final judgment unit 54 stores the result of the final judgment in the storage unit 60 in association with the moving image data Da. In even more particular, the final judgment unit 54 synchronizes the result of the final judgment with the moving image data Da. The final judgment unit 54 associates the result of the final judgment with each frame image data on which the final judgment was based.

The association may be based on the identifier of the frame image data, or on the elapsed time from the start of playback of the video image data Da.

(b5. Display control unit 70)
The display control unit 70 controls the display of the display unit 80. The display control unit 70 displays the result of the final judgment together with the moving image data Da based on a user operation. Due to the association described above, the result of the final judgment changes successively as the playback of the moving image data Da progresses. In this example, the result of the final judgment is updated every second during playback of the moving image data Da.

<C-trained model>
The trained models M20, M51, M52, and M53 will be explained.

The trained model M20 is generated by a training dataset and a training program that have been prepared in advance. The training dataset includes multiple training data. Each training data is image data (still image data, frame image data) of a worker performing work in a workplace, to which label information (correct answer data) indicating the person area in the image data has been added. In this example, the person area in the label information is specified as a rectangular area.

Like trained model M20, trained model M51 is generated from a training data set and a training program that have been prepared in advance. The training data set includes multiple pieces of training data. Each piece of training data is image data of a worker performing welding work in a workplace, to which label information (correct answer data) indicating that the work type is welding has been added.

Like trained models M20 and M51, trained model M52 is generated from a training data set and a training program that have been prepared in advance. The training data set includes multiple pieces of training data. Each piece of training data is image data of a worker performing grinding work in a workshop, to which label information indicating that the work type is grinding has been added.

Like trained models M20, M51, and M52, trained model M53 is generated from a training data set and a training program that have been prepared in advance. The training data set includes multiple pieces of training data. Each piece of training data is image data of a worker performing gouging work in a workplace, to which label information indicating that the work type is gouging has been added.

The trained models M20, M51, M52, and M53 are networks classified as DNNs (Deep Neural Networks). The trained models M20, M51, M52, and M53 include a preprocessing network classified as a CNN (Convolutional Neural Network), an intermediate layer, an activation function corresponding to the output layer, and a Softmax function.

The preprocessing network is expected to function as a kind of filter for extracting effective features for calculating estimation results from features with relatively large degrees. The preprocessing network has a configuration in which convolutional layers (CONV) and pooling layers (Pooling) are arranged alternately. Note that the number of convolutional layers and pooling layers does not have to be the same, and an activation function such as ReLU (rectified linear unit) is arranged on the output side of the convolutional layer.

More specifically, the preprocessing network is constructed to receive feature inputs and output internal features that indicate specified attribute information. The intermediate layer is made up of a fully connected network with a specified number of layers, and sequentially connects the outputs from the preprocessing network for each node using weights and biases determined for each node.

An activation function such as ReLU is placed on the output side of the intermediate layer, and finally, the estimation result is output after being normalized to a probability distribution using the Softmax function.

When the learning program optimizes the parameter values, any optimization algorithm can be used. More specifically, the optimization algorithm can be, for example, a gradient method such as SGD (Stochastic Gradient Descent), Momentum SGD (SGD with inertia term added), AdaGrad, RMSprop, AdaDelta, or Adam (Adaptive moment estimation).

D. Processing Flow
The above-mentioned process flow in the information processing device 2 will be further described with reference to a flow chart or the like.

Figure 4 is a flow diagram showing the flow of the determination process.

As shown in FIG. 4, in step S1, the information processing device 2 detects a person area in each frame image data #N constituting the moving image data Da. In step S2, the information processing device 2 uses the person area detection result to determine the task type for each frame image data #N, and further performs a final determination every second based on the determination result for each frame image data #N.

The processes of steps S1 and S2 are executed by the processor 201. The process of step S1 is executed by the person area detection unit 20 (FIG. 3). Specifically, the process of step S1 is realized by the trained model M20. The process of step S2 is executed by the task type determination unit 50 (FIG. 3). Specifically, the process of step S2 is realized by the trained models M51, M52, and M53.

Figure 5 is a flow diagram showing the details of the processing of step S1 in Figure 4.

As shown in FIG. 5, in step S101, the processor 201 accepts a setting input of the number of people to be determined via the input device 204. Typically, the processor 201 accepts an input indicating one person (single person work) or two people (two people work). The reason for accepting a setting input of the number of people is to improve the accuracy of detecting the person area.

In step S102, the processor 201 reads the video data Da from the memory 202. In step S103, the processor 201 reads the trained model M20 for human area detection from the memory 202. In step S104, the processor 201 executes human area detection processing using the trained model M20 for each frame image data #N.

In step S105, the processor 201 adjusts the judgment according to the number of people set in step S101. Specifically, the processor 201 selects person areas for the set number of people in the same frame image data in descending order of the accuracy of the judgment of the person area. For example, when the set number of people is one, the processor 201 selects the person area with the highest accuracy from multiple person areas (candidate areas). When the set number of people is two, the processor 201 selects the person area with the highest accuracy and the person area with the second highest accuracy from multiple person areas (candidate areas). In step S106, the processor 201 performs post-processing such as front and back completion.

In step S107, the processor 201 stores the person area detection result (coordinates) in a predetermined format in the memory 202 for each frame image data #N. The processor 201 typically stores the person area detection result in one of the highly compatible file formats (for example, the csv (Comma Separated Value) format). In detail, the processor 201 determines the person area detection (coordinates) in association with information such as a timestamp, the frame number of the frame image data, the object number, and the accuracy of the determination of the person area detection.

In step S108, the processor 201 creates video data Dc of the person area detection result and stores it in the memory 202. Note that the video data Dc is obtained by superimposing a figure (rectangle) indicating the person area on the video data Da.

Figure 6 is a flow diagram showing the details of the processing of step S2 in Figure 4.

As shown in FIG. 6, in step S201, the processor 201 reads video data Da from the memory 202. In step S202, the processor 201 creates frame difference video data Db using a frame difference method. This process is executed by the generator 30 (FIG. 3). Details of the process of step S202 will be described later (FIG. 7).

In step S203, the processor 201 reads the person area detection result from the memory 202. Note that the person area detection result includes information such as the coordinates of the person area and the identifier (timestamp or frame number) of the frame image data described above.

In step S204, the processor 201 extracts an area corresponding to a person area from each frame image data P constituting the frame difference moving image data Db based on the person area detection result. In other words, the processor 201 cuts out an image. In more detail, the processor 201 uses the person area detection result associated with each frame image data P to extract an area corresponding to a person area from each frame image data P. In more detail, the processor 201 cuts out an image using a different person area detection result for each frame image data P.

In step S205, the processor 201 reads from the memory 202 the trained models M51, M52, and M53 for determining the work type. In step S206, the processor 201 executes the trained models M51, M52, and M53 to perform a process of determining the work type from each image data Q (image data of the cut-out portion) extracted in step S204. Details of the process in step S206 will be described later (FIG. 9).

In step S207, the processor 201 outputs the work type determination result in step S206 as the final determination result for each second. Specifically, as described above, the processor 201 determines (final determination) that the most numerous work type in each second is the work being performed in the workplace during that second. The final determination results for each second are temporarily stored in the working area of the memory 202 (typically, the RAM 222) one by one.

In step S208, if the processor 201 determines that the work performed during one second cannot be classified, the processor 201 executes a movement determination process to determine whether the work is a movement of a worker. Details of the movement determination process will be described later.

In step S209, the processor 201 executes post-processing such as completion of beginning and end. In step S210, the processor 201 stores the final judgment result for each second in a non-volatile manner in the memory 202 in a predetermined format. Typically, the processor 201 stores the final judgment result in the SSD 223 or the HDD 224 as data in a csv format. In more detail, the processor 201 judges the final judgment result in association with information such as a timestamp and a frame number.

In step S211, the processor 201 creates video data Dd including the final judgment result and stores it in the memory 202. Note that the video data Dd is video data Da with an image superimposed thereon indicating the final judgment result using identification information such as characters.

When the user 950 plays the video data Dd on the information processing device 2, the display 203 displays the work type superimposed on the image of the work being performed in the workplace. The display of the work type is updated every second.

Next, the movement determination process of step S208 will be described. In the movement determination process, the processor 201 first calculates the average value of the width and height of each detected person area. In other words, the processor 201 calculates the position of the center of gravity of each person area within frame image data #N.

Next, the processor 201 calculates the average value of the center of gravity position for each second. After that, the processor 201 associates each average value of the center of gravity position with the final determination result at the same time (timing). Furthermore, the processor 201 calculates the amount of movement of the person area for each second based on each average value of the center of gravity position.

The processor 201 extracts the final judgment result that is determined to be other work (unclassifiable) from the working area of the memory 202. The processor 201 then determines whether or not the movement amount associated with the extracted final judgment result is within a predetermined range (between a lower threshold and an upper threshold). If the movement amount is within the predetermined range, the processor 201 replaces the information of other work (unclassifiable) with "movement".

Figure 7 is a flow diagram for explaining the details of the process of step S202 in Figure 6. Figure 8 is a diagram for explaining the process of Figure 7 using image data.

As shown in FIG. 7, in step S2201, the processor 201 sets the value of the above-mentioned variable N (#N) to 2. In step S2202, the processor 201 acquires two consecutive frame image data #N-1 and #N from the loaded video image data Da. FIG. 8 shows examples of frame image data #N-1 and frame image data #N.

In step S2203, the processor 201 generates differential image data R (see FIG. 8) that represents the difference between frame image data #N-1 and frame image data #N. In step S2204, the processor 201 binarizes the differential image data R to generate binarized image data T (see FIG. 8).

In step S2205, the processor 201 performs a closing process on the binarized image data T to generate closing image data U (see FIG. 8). The closing image data U is used as a mask image. In step S2206, the processor 201 performs a masking process on the frame image data #N using the closing image data U. As a result, image data V after the mask process (see FIG. 8) is generated.

In step S2207, the processor 201 stores the image data V after the mask processing in the memory 202 as new image data #N-1. In step S2208, the processor 201 determines whether the moving image data Da is finished. Specifically, the processor 201 determines whether the above-mentioned processing has been performed on all two consecutive frame image data of the moving image data Da.

If it is determined that the video data Da is not complete (NO in step S2208), the processor 201 increments the value of N by 1 in step S2210. If it is determined that the video data Da is complete (YES in step S2208), the processor 201 generates the above-mentioned frame difference video data Db by connecting all of the new image data #N in chronological order in step S2209, and stores it in the memory 202.

After step S2209, the processor 201 proceeds to step S203 in FIG. 6.

Figure 9 is a flow diagram for explaining the details of the processing of step S206 in Figure 6.

As shown in FIG. 9, in step S2601, the processor 201 reads each image data Q (image data of the cut-out portion) extracted from the memory 202.

In step S2602, the processor 201 performs a judgment on one image data Q using each of the trained models M51, M52, and M53. Specifically, the processor 201 executes each of the trained models M51, M52, and M53 to calculate the accuracy of each operation (welding operation, grinding operation, and gouging operation) as described above. In step S2603, the processor 201 determines whether the maximum accuracy among the calculated accuracy of the three operations exceeds a threshold value (e.g., 0.6).

If it is determined that the maximum accuracy exceeds the threshold (YES in step S2603), the processor 201 adopts the judgment of the maximum accuracy in step S2604. As a specific example, if the accuracy output from the trained model M51 is 0.7, the accuracy output from the trained model M52 is 0.1, and the accuracy output from the trained model M53 is 0.05, the processor 201 adopts the judgment by the trained model M51. The processor 201 determines that the work type is welding.

If it is determined that the maximum accuracy does not exceed the threshold (NO in step S2603), the processor 201 determines in step S2607 that classification is not possible. In step S2605, the processor 201 stores the result of the determination in the memory 202.

In step S2606, the processor 201 determines whether or not the image data Q is the last. If it is determined that the image data Q is the last (YES in step S2606), the processor 201 ends the series of processes in step S206 and proceeds to step S207 in FIG. 6. If it is determined that the image data Q is not the last (NO in step S2606), the processor 201 switches the processing target to the next image data Q in step S2608. The processor 201 then proceeds to step S2602.

<E. Final judgment result example>
FIG. 10 is a diagram showing data including the final determination results stored in the memory 202 in a highly compatible file format (in this example, the csv format).

As shown in Figure 10, the "Predict" column of the data records the final judgment results every second. "Indistinguishable" means "unclassifiable," "Moving" means "moving," and "Welding" means "welding."

Note that "G_X" and "G_Y" respectively represent the X and Y coordinates of the center of gravity of the person area. More specifically, "G_X" and "G_Y" are the average values of the center of gravity position over one second. "Width" and "Height" respectively represent the width and height of the person area.

In this way, the information processing device 2 identifies the task type every second. Therefore, by accumulating the time for each task type, the time required for each task type can be calculated. The information processing device 2 calculates such times, for example, in response to a user operation, and displays the calculated results (task time for each task type) on the display 203.

<F. Summary>
Some of the processes executed by the information processing device 2 can be summarized as follows.

(1) The information processing device 2 includes a video data acquisition unit 10 that acquires a series of multiple frame image data (video data, image data) captured by a camera 1 (visible light camera). The camera 1 is fixed in its installation position and orientation, and captures an image of a workplace where workers are performing work that involves blinking light.

The information processing device 2 further includes a person area detection unit 20 that detects the area of the worker (person area) in frame image data #N of the plurality of frame image data, a generation unit 30 that generates image data showing a change in the state of the subject based on the frame image data #N and frame image data #N-1 that precedes frame image data #N of the plurality of frame image data, an extraction unit 40 that extracts image data of an area corresponding to the detected area of the worker from the generated image data, and a work type determination unit 50 that determines the type of work being performed in the workplace based on the extracted image data.

According to the information processing device 2 configured in this way, it is possible to extract image data that indicates a change in state in the worker's area from a plurality of frame image data obtained by imaging with a visible light camera. Furthermore, according to the information processing device 2, the type of work being performed in the work area is determined based on the image data that indicates the change in state in the worker's area.

Therefore, the information processing device 2 can determine the type of work being performed in the workplace based on video image data captured by a visible light camera. In particular, the type of work that involves blinking of light can be accurately determined for each frame of image data.

(2) The information processing device 2 further includes a storage unit 60 that stores information indicating the type of work that has been determined. With this configuration, the information processing device 2 stores the results of the determination, and can therefore perform various post-processing steps using the results (for example, the above-mentioned final determination process and display process).

(3) The generation unit 30 generates image data indicating a change in the state of the subject by a frame difference method using frame image data #N and frame image data #N-1. With this configuration, image data indicating a change in the state of the subject can be generated by using the frame difference method, which is one of the methods for detecting moving objects.

(4) The work type determination unit 50 determines whether the work being performed in the workplace is welding work. With this configuration, it is possible to determine whether work accompanied by blinking light is welding work.

(5) The task type determination unit 50 determines which of multiple pre-specified tasks the task being performed in the workplace is. With this configuration, it is possible to determine which of the multiple tasks the task that is accompanied by blinking light is.

(6) The multiple operations include welding, grinding, and gouging. With this configuration, it is possible to determine whether the operation accompanied by the blinking of light is welding, grinding, or gouging.

(7) The work type determination unit 50 includes a trained model M51 that receives the extracted image data as input and determines whether the work being performed in the workplace is welding work, a trained model M52 that receives the extracted image data as input and determines whether the work being performed in the workplace is grinding work, and a trained model M53 that receives the extracted image data as input and determines whether the work being performed in the workplace is gouging work.

The work type determination unit 50 determines whether the work being performed in the workplace is welding work, grinding work, or gouging work, based on the results of the determination by the trained model M51, the results of the determination by the trained model M52, and the results of the determination by the trained model M53.

According to this configuration, the information processing device 2 uses the trained models M51, M52, and M53 to determine the task type. Therefore, the information processing device 2 can make a more accurate determination than a rule-based determination process that does not use trained models.

(8) The person area detection unit 20 is a trained model M20 that receives frame image data #N as input and outputs information indicating the area of the worker. With this configuration, the information processing device 2 uses the trained model M20 to detect the area of the worker (person area). Therefore, the information processing device 2 can achieve more accurate detection than a rule-based detection process that does not use a trained model.

(9) For each of the multiple frame image data, the information processing device 2 performs detection by the person area detection unit 20, generation by the generation unit 30, extraction by the extraction unit 40, and judgment by the task type determination unit 50. The information processing device 2 calculates the work time for each task type based on the number of judgments for each task type determined by the task type determination unit 50.

With this configuration, the information processing device 2 calculates the total task time for each task based on multiple frame image data captured by the camera 1. Therefore, the user 950 of the information processing device 2 can know how much time each task takes.

G. Modifications
(1) Fig. 11 is a flow chart showing a modified example of the series of processes in step S202 shown in Fig. 7. In the following, a configuration using a frame difference method using three consecutive frame image data will be described.

Referring to FIG. 11, the series of processes shown in FIG. 11 differs from the series of processes shown in FIG. 7 in the following respects. The series of processes shown in FIG. 11 includes steps S2202A, S2203A, S2204A, and S2205A instead of steps S2202, S2203, S2204, and S2205 (see FIG. 7). Furthermore, the series of processes shown in FIG. 11 differs from FIG. 7 in that it includes step S2211, which does not include this step.

Note that the processing of the other steps in FIG. 11 is the same as the processing described in FIG. 7. Therefore, the following describes these steps S2202A, S2203A, S2204A, S2205A, and S2211.

In step S2202A after step S2201, the processor 201 obtains three consecutive frame image data #N-1, #N, and #N+1 from the loaded video image data Da. In step S2203A, the processor 201 generates differential image data R representing the difference between frame image data #N-1 and frame image data #N, and differential image data R representing the difference between frame image data #N and frame image data #N+1.

In step S2204A, the processor 201 binarizes each differential image data R to generate two binary image data T. In step S2211, the processor 201 extracts the shared portion of the two binary image data T to generate image data W. Specifically, the processor 201 sets the portions of the two binary image data T that are both white (i.e., portions with a value of 1) to white (value 1), and sets the rest to black (value 0).

In step S2205A, the processor 201 performs a closing process on the extracted image data W to generate closing image data U. The processor 201 then advances the process to step S2206.

In more detail, in this modified example, the generation unit 30 generates image data indicating a change in the state of the subject by a frame difference method using frame image data #N, frame image data #N-1, and frame image data #N+1.

This type of processing enables more accurate determination processing than the configuration in Figure 7.

(2) In the above, an example was described in which the generation unit 30 generates image data indicating a change in the subject's condition using successive frame image data #N and #N-1, but this is not necessarily limited to this. The generation unit 30 may be configured to generate image data indicating a change in the subject's condition based on frame image data #N and a predetermined number of frame image data (one or more frames) before frame image data #N. For example, the generation unit 30 may generate image data indicating a change in the subject's condition based on frame image data #N and frame image data #N-2, which is two frames before frame image data #N.

(3) If it is possible to generate image data that indicates changes in the state of a subject, it is possible to apply moving object detection methods other than the frame difference method.

(4) It is not necessary to use a trained model to detect the worker's area (person area). The worker's area may be detected using a rule-based method.

The embodiments disclosed herein are illustrative and are not limited to the above. The scope of the present invention is defined by the claims, and is intended to include all modifications within the scope and meaning equivalent to the claims.

1 camera, 2 information processing device, 10 video data acquisition unit, 20 person area detection unit, 30 generation unit, 40 extraction unit, 50 work type determination unit, 51 welding work determination unit, 52 grinding work determination unit, 53 gouging work determination unit, 54 final determination unit, 60 storage unit, 70 display control unit, 80 display unit, 201 processor, 202 memory, 203 display, 204 input device, 205 communication interface, 206 card reader, 207 port, 900 worker, 950 user, 1000 determination system, M20, M51, M52, M53 trained model.

Claims (6)

an acquisition means for acquiring a plurality of consecutive frame image data obtained by imaging with a visible light camera, the visible light camera being installed at a fixed position and attitude, and capturing an image of a workplace where a worker is performing work involving blinking light as a subject;
a detection means for detecting an area of the worker in a first frame image data of the plurality of frame image data;
a generating means for generating difference data between the first frame image data and a second frame image data that is a predetermined number of frames before the first frame image data among the plurality of frame image data by a frame difference method, and for generating image data that indicates a change in state of the subject based on the generated difference data and the first frame image data;
an extraction means for extracting image data of an area corresponding to an area of the detected worker from the generated image data;
The information processing device further comprises a determination means for determining whether a type of work being performed at the workplace is welding work, grinding work, or gouging work, based on the extracted image data. The information processing device according to claim 1, wherein the second frame image data is the frame image data immediately preceding the first frame image data. The determination means is
A first trained model that receives the extracted image data as an input and determines whether the work being performed at the workplace is the welding work; and
A second trained model that receives the extracted image data as an input and determines whether the work being performed at the workplace is the grinding work; and
a third trained model that receives the extracted image data as an input and determines whether the work being performed at the work site is the gouging work;
3. The information processing device according to claim 1 or 2, which determines whether the work being performed in the workplace is the welding work, the grinding work, or the gouging work based on a result of judgment using the first trained model, a result of judgment using the second trained model, and a result of judgment using the third trained model. The information processing device according to claim 1 , wherein the detection means is a fourth trained model that receives the first frame image data as an input and outputs information indicating an area of the worker. For each of the plurality of frame image data, detection by the detection means, generation by the generation means, extraction by the extraction means, and determination by the determination means are performed;
The information processing apparatus according to claim 1 , further comprising: a determining unit that determines a task time for each of the task types based on a determination number for each of the task types determined by the determining unit. a step of capturing an image of a workplace where a worker is performing work involving blinking light by a visible light camera as a subject, the visible light camera having a fixed installation position and attitude;
acquiring a plurality of consecutive frame image data captured by the visible light camera;
detecting an area of the worker in a first frame image data of the plurality of frame image data;
generating difference data between the first frame image data and a second frame image data that is a predetermined number of frames before the first frame image data among the plurality of frame image data by a frame difference method;
generating image data representing a change in a state of the subject based on the generated difference data and the first frame image data;
extracting image data of an area corresponding to an area of the detected worker from the generated image data;
The information processing method further comprises a step of determining whether a type of work being performed at the workplace is welding work, grinding work, or gouging work based on the extracted image data.

Images