JP7572843B2 - Worker management support system and worker management support processing method - Google Patents

Description

This disclosure relates to technology for managing the status of employees based on images of employees working in a company or the like.

In recent years, research has been conducted into technology for acquiring vital sign information (hereinafter referred to as vital information) that indicates a person's mental and physical state from a video of the person (see Non-Patent Document 1). Examples of vital information include heart rate, blood pressure, body temperature, blood oxygen concentration, stress level, and blood alcohol concentration. This technology makes it possible to acquire highly accurate vital information from images captured by a camera without using dedicated devices such as sensing wear worn to measure a person's heart rate and body surface temperature, or a pulse oximeter that measures a person's blood oxygen concentration by pinching a finger.

Makoto Yoshizawa, Cyberscience Center, Tohoku University, Norihiro Sugita, Graduate School of Engineering, Tohoku University, "Development of the "Magic Mirror" Blood Circulation Monitoring Device," Optical Technology Contact, October 2017 issue

People who work in any kind of job (workers), such as employees working in manufacturing sites or warehouses, drivers of trains, buses, taxis, etc., and pilots of airplanes or ships, need to be physically and mentally healthy in order to perform their jobs safely. Non-Patent Document 1 discloses a mirror-type display in which the technology disclosed therein is applied to a computer integrated with a workplace mirror. This makes it possible to check the health status of employees in the workplace. However, with the technology disclosed in Non-Patent Document 1, it is not possible to check the health status of workers unless they stand in front of the mirror-type display, and it is not possible to check the work of employees working in manufacturing sites or warehouses in real time.

One objective of this disclosure is to provide technology that assists employees in maintaining and managing the work they perform.

A worker management support system according to one aspect of the present disclosure includes a camera that captures video of a worker engaged in work, a processing device that predicts the occurrence of a predetermined event related to the worker based on vital sign information of the worker obtained from the captured video, and a management device that indicates that the occurrence of the predetermined event has been predicted. The processing device learns learning data to generate an event prediction model that takes the vital sign information of the worker as input and outputs a predicted result of the occurrence of a predetermined event related to the worker, and predicts the occurrence of the predetermined event related to the worker using the event prediction model. The management device obtains performance information from the worker indicating whether the predicted predetermined event has occurred, and provides the performance information to the processing device as the learning data.

According to one aspect of the present disclosure, it is possible to assist in maintaining and managing the mental and physical condition of workers.

FIG. 1 is a block diagram of a worker management system according to an embodiment. 2 is a diagram showing an example of the configuration of a data storage area of the moving image shooting device shown in FIG. 1 . FIG. 2 is a diagram illustrating an example of a configuration of an AI server in the cloud system shown in FIG. 2 is a diagram illustrating an example of a configuration of a data storage area of the cloud system illustrated in FIG. 1 . 2 is a diagram illustrating an example of a configuration of a data storage area of the cloud system illustrated in FIG. 1 . 2 is a diagram showing an example of an attribute information data storage area of a worker in the cloud system shown in FIG. 1 . 4 is a flowchart for explaining a process in the video shooting device of the worker management system shown in FIG. 1 . 4 is a flowchart for explaining a process performed when a cloud system acquires information managed in a business operator system of the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a worker behavior analysis process in the cloud system of the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a process for predicting an illness of a worker without using AI in the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a process performed when a malfunction prediction is notified to a business operator system from a cloud system of the worker management system shown in FIG. 1 . 4 is a flowchart for explaining a process for creating a mental and physical abnormality prediction model in the worker management system shown in FIG. 1 . 4 is a flowchart for explaining a process for creating a work defect prediction model in the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a process for generating advice according to vital information and motion information of a worker in the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a process for detecting mental and physical abnormalities of a worker using AI in the worker management system shown in FIG. 1 . 2 is a flowchart for explaining a process for predicting work defects or mistakes using AI in the worker management system shown in FIG. 1 . 4 is a flowchart for explaining a process of checking vital sign information and the like of a worker in a business operator system of the worker management system shown in FIG. 1 . 10 is a flowchart for explaining a process of push-distributing important information about work to workers in the worker management system shown in FIG. 1 . 2 is a flowchart for explaining a process in which a worker checks vital signs trends and the like using his or her own mobile terminal in the worker management system shown in FIG. 1 .

Below, an embodiment of the present invention will be described with reference to the drawings.

Figure 1 is a block diagram of the worker management system according to this embodiment.

As shown in FIG. 1, the worker management system includes a video capture device 10, a cloud system 20, a mobile terminal 50, and a business management system 60. The video capture device 10, the mobile terminal 50, and the business management system 60 are each configured to be able to communicate with the cloud system 20.

Video recording device 10 is the recording device in the present invention, and is installed as a so-called fixed camera in a work area where workers perform their work. Video recording device 10 records videos of workers engaged in work, and has a camera 11, a CPU 12, and a memory 13. CPU 12 realizes a face authentication control unit 14, a video vitals control unit 15, a behavior analysis control unit 16, a cloud communication control unit 17, and a camera image control unit 18. Memory 13 is provided with a data storage area 19 in which data is stored by camera image control unit 18.

The camera 11 captures video including workers at the work site where it is installed.

The face authentication control unit 14 recognizes the face of the worker from the video captured by the camera 11 and performs face authentication. The video vitals control unit 15 acquires vital information of the worker based on the video captured by the camera 11. The behavior analysis control unit 16 acquires motion information indicating the behavior of the worker by motion capture based on the video captured by the camera 11. The behavior analysis control unit may be included in the cloud system 20 instead of the video capture device 10, or may be included in both. The camera image control unit 18 stores the video captured by the camera 11 in the data storage area 19 together with the vital information of the worker acquired by the video vitals control unit 15. The motion information of the worker acquired by the behavior analysis control unit 16 may also be stored in the data storage area 19. The cloud communication control unit 17 transmits the face information authenticated by the face authentication control unit 14 and the vital information acquired by the video vitals control unit 15 to the cloud system 20 via a communication line (not shown). The worker's movement information acquired by the behavior analysis control unit 16 may also be transmitted to the cloud system 20.

Figure 2 is a diagram showing an example of the configuration of the data storage area 19 of the video recording device 10 shown in Figure 1.

As shown in FIG. 2, images captured by the camera 11 are registered as video files and still image files together with the capture date and time in the data storage area 19 of the video capture device 10 shown in FIG. 1. In addition, the video files and still image files are sent to the cloud system 20 together with information on the capture date and time.

The cloud system 20 is a processing device and is realized on the cloud. The cloud system 20 configures an AI server 40 with a computing system including a CPU 22 and a memory 23, and predicts the occurrence of a predetermined event related to the worker by the AI server 40 based on the vital information of the worker obtained from the video taken by the video shooting device 10. Specifically, by learning the learning data, an event prediction model is generated in which the vital information of the worker is input and the predicted result of the occurrence of a predetermined event related to the worker is output, and the occurrence of a predetermined event related to the worker is predicted using the generated event prediction model. The CPU 22 realizes the video shooting device communication control unit 21, the behavior analysis control unit 25, the labor management control unit 28, the information provision control unit 29, the worker communication control unit 30, and the business communication control unit 32. Details of the AI server 40 will be described later. The memory 23 includes a data storage area 26 in which data received from the video capture device 10 and data acquired by the behavior analysis control unit 25 are stored, worker attribute information 27, and business operator system production management information 32.

The video shooting device communication control unit 21 receives data transmitted from the video shooting device 10 via a communication line (not shown). The face authentication control unit 24 recognizes the face of the worker from the data transmitted from the video shooting device 10 and performs face authentication. The behavior analysis control unit 25 acquires motion information indicating the behavior of the worker by motion capture based on the data transmitted from the video shooting device 10. The labor management control unit 28 checks the vital information of the worker by comparing the numerical value of the vital information transmitted from the video shooting device 10 with a predetermined threshold value. The information provision control unit 29 provides the business system 60 with events related to the mental and physical state of the worker in response to a request from the mobile terminal 50 or the business system 60, or autonomously, in response to the prediction result in the AI server 40. The worker communication control unit 30 transmits the information provided by the information provision control unit 29 to the mobile terminal 50 via a communication line (not shown) in response to a request from the mobile terminal 50 or autonomously. The carrier communication control unit 32 transmits information provided by the information provision control unit 29 and carrier system management information 31 to the carrier system 60 via a communication line (not shown), either in response to a request from the carrier system 60 or autonomously.

Figure 3 is a diagram showing an example configuration of the AI server 40 of the cloud system 20 shown in Figure 1.

As shown in FIG. 3, the AI server 40 of the cloud system 20 shown in FIG. 1 has a first mental and physical abnormality prediction unit 41a, a second mental and physical abnormality prediction unit 41b, a work defect prediction unit 42, a prediction control unit 44, learning units 45 and 46, and an advice unit 47. In addition, the second mental and physical abnormality prediction unit 41b has a mental and physical abnormality prediction model 41c, the work defect prediction unit 42 has a work defect prediction model 42a, and the advice unit 47 has a database 48.

The first mental and physical abnormality prediction unit 41a predicts the occurrence of mental and physical abnormalities in the worker based on the result of comparing the index value of the vital information with a predetermined threshold value and the movement information. The second mental and physical abnormality prediction unit 41b inputs the vital information and movement information to the mental and physical abnormality prediction model 41c, and obtains the result of predicting the occurrence of mental and physical abnormalities in the worker from the output of the mental and physical abnormality prediction model 41c. The prediction control unit 44 sets the prediction accuracy of the mental and physical abnormality prediction model 41c based on the performance information, and if the prediction accuracy is set to good, provides the prediction result of the second mental and physical abnormality prediction unit 41b to the business operator system 60, and if the prediction accuracy is set to poor, provides the prediction result of the first mental and physical abnormality prediction unit 41a to the business operator system 60.

The mental and physical abnormality prediction model 41c receives the worker's vital signs and movement information as input and outputs the predicted occurrence of mental and physical abnormalities in the worker, and is included in the event prediction model.

The work defect prediction unit 42 inputs the vital sign information to the work defect prediction model 42a, and obtains the result of the prediction of the occurrence of work defects by the worker from the output of the work defect prediction model 42a. Note that the work defect prediction model 42a inputs the vital sign information and movement information of the worker, and outputs the prediction result of the occurrence of work defects by the worker, and is included in the event prediction model.

The learning units 45 and 46 each generate a mental and physical abnormality prediction model 41c and a work defect prediction model 42a by learning actual performance information indicating whether or not a predicted event has actually occurred as learning data.

The advice unit 47 generates an advice database 48 that stores vital information and movement information in association with work quality, acquires vital information and movement information corresponding to a specified work quality from the advice database 48, and generates work-related advice for the worker based on the difference between the acquired vital information and movement information and the worker's current vital information and movement information.

Figures 4 and 5 are diagrams showing an example configuration of the data storage area 26 of the cloud system 20 shown in Figure 1.

As shown in FIG. 4, the data storage area 26 of the cloud system 20 shown in FIG. 1 stores the vital information of workers sent from the video capture device 10 and image files of still images, linked to the motion capture device ID of the video capture device that sent the video, vital information, and still images and the date and time they were received. The vital information stored here includes the vital information of all workers who appeared in the video. The vital information stored includes the heart rate, blood oxygen concentration, blood pressure, body temperature, stress level, and blood alcohol concentration of each worker, linked to the worker ID.

As shown in FIG. 5, the data storage area 26 stores the motion information of the worker, which is the analysis result in the motion analysis control unit 16 or the motion analysis control unit 25. The motion information and video files are stored in association with the motion capture device ID of the video capture device that transmitted the video, vital information, and still images, and the date and time when they were received. The motion information stores the X, Y, and Z coordinates indicating the position of each measurement point indicating each joint or body part of the worker in chronological order.

Figure 6 is a diagram showing an example of the worker attribute information data storage area 26 of the cloud system 20 shown in Figure 1.

As shown in FIG. 6, the worker attribute information data storage area 26 of the cloud system 20 shown in FIG. 1 stores the employee ID, name, gender, date of birth, and medical history of the workers managed by the system.

The mobile terminal 50 has a cloud communication control unit 51 and an information display unit 52. The cloud communication control unit 51 requests the worker's tendencies from the cloud system 20 in response to the worker's operation. The cloud communication control unit 51 also receives the worker's tendencies and vital sign information that are sent from the cloud system 20 in response to the worker's request or that are sent autonomously from the cloud system 20. The information display unit 52 displays the worker's tendencies received by the cloud communication control unit 51 on a display.

The business operator system 60 is a management device that indicates that a specified event is predicted to occur. The business operator system 60 has a cloud communication control unit 61, a data storage area 62, a labor status management unit 63, and a business management system 64, and acquires performance information from workers indicating whether a specified event predicted by the cloud system 20 has occurred, and provides the acquired performance information to the cloud system 20 as learning data.

The cloud communication control unit 61 requests the cloud system 20 to send vital sign information and images, or receives these, receives predictions of the occurrence of mental and physical disorders of workers, and transmits performance information indicating whether or not the predicted events have actually occurred. The data storage area 62 stores information received by the cloud communication control unit 61 and information managed by the labor status management unit 63. The labor status management unit 63 requests the cloud system 20 to send vital sign information and images, displays vital sign information sent from the cloud system 20 and stored in the data storage area 42, and collects performance information indicating whether or not the predicted events have actually occurred. The work management system 64 manages the work of workers.

The following describes the processing performed in the worker management system configured as above.

First, we will explain the processing in the video capture device 10.

Figure 7 is a flowchart for explaining the processing in the video capture device 10 of the worker management system shown in Figure 1.

When a video of a worker is captured by the camera 11 of the video capture device 10 (step S101), the face recognition control unit 14 first performs face recognition of the worker from the video captured by the camera 11 (step S102). In addition, the video vitals control unit 15 acquires vital signs, which are vital information of the worker, based on the video captured by the camera 11 (step S103).

When the vital sign value is acquired, the video vital sign control unit 15 compares the acquired vital sign value with a predetermined threshold value and determines whether the acquired vital sign value exceeds the threshold value (step S104).

If the acquired vital signs do not exceed the threshold, a determination is made as to whether the worker's behavior is abnormal based on the video or the motion information analyzed by the behavior analysis control unit 16 (step S105).

If the video vitals control unit 15 determines that the acquired vital signs value exceeds a threshold value or that the worker's behavior is abnormal, the camera image control unit 18 instructs the worker to save the video (step S106), and the video of the worker's surroundings captured by the camera 11 is saved in the data storage area 19 (step S107). A still image captured at the same time is also saved in the data storage area 19.

Furthermore, under the control of the cloud communication control unit 17, the vital information acquired by the video/vital control unit 15 is transmitted to the cloud system 20 together with the video and still images captured by the camera 11 (step S108). When the behavior analysis control unit 16 acquires the motion information of the worker, the motion information is also transmitted to the cloud system 20.

Next, we will explain the process by which the cloud system 20 acquires information managed in the business operator system 60.

Figure 8 is a flowchart to explain the process when the cloud system 20 acquires information managed in the operator system 60 of the worker management system shown in Figure 1.

In the business operator system 60, the business management system 64 stores information on the work process and work defects related to the workers (step S201). In addition, the labor status management unit 64 stores the past occurrence of mental and physical abnormalities for each worker (step S202).

The process information, work defect information, and past occurrences of mental and physical abnormalities are transmitted to the cloud system 20 under the control of the cloud communication control unit 61 (step S203).

The process information, work defect information, and past occurrences of mental and physical abnormalities sent from the business operator system 60 are received by the business operator communication control unit 32 of the cloud system 20 (step S204).

The received process information, work defect information, and past occurrences of mental and physical abnormalities are then stored as business operator system production management information 31 (step S205), and are then used as learning data in the AI server 40 to generate a mental and physical abnormality prediction model and a work defect model (step S206).

Next, we will explain the behavior analysis process of workers in the cloud system 20.

Figure 9 is a flowchart for explaining the worker behavior analysis process in the cloud system 20 of the worker management system shown in Figure 1.

When the vital information and video transmitted from the video shooting device 10 are received by the video shooting device communication control unit 21 of the cloud system 20 (step S301), the face recognition control unit 24 in the cloud system 20 first performs face recognition of the worker using the received video (step S302). In this face recognition, the worker is identified by comparing the face data of the worker stored in the worker attribute information 27.

Next, the received vital information and video are stored in the data storage area 26 (step S303).

Next, the behavior analysis control unit 25 uses motion capture to convert the movements of the worker in the received video into data (step S304), identifies the worker by referring to the worker's attribute information 27, and stores the movement information in the data storage area 26 in association with the worker (step S305), and returns to the processing of step S301.

Next, we will explain the process of predicting worker ailments without using AI.

Figure 10 is a flowchart to explain the process of predicting worker ailments without using AI in the worker management system shown in Figure 1.

First, the labor management control unit 28 of the cloud system 20 checks the vital information stored in the data storage area 26 in step S305 of FIG. 9 (step S401). At this time, the worker's attribute information 27 is referenced to identify the worker.

If the vital sign value based on the vital sign information stored in the data storage area 26 exceeds a predetermined threshold value (step S402), the illness flag for that worker is set to "ON" (step S403). Possible vital signs include heart rate, blood oxygen concentration, blood pressure, body temperature, stress level, and blood alcohol concentration, and a combination of these values may be judged against threshold values.

In addition, if the vital signs based on the vital signs information stored in the data storage area 26 do not exceed a predetermined threshold value, or after the illness flag is set to "ON", the labor management control unit 28 checks whether the worker's behavior is abnormal (steps S404, S405).

If the worker's behavior is determined to be abnormal, the behavior abnormality flag for that worker is set to "ON" (step S406).

Then, the two flags are checked (step S407), and if at least one of them is "ON", the operator communication control unit 32 controls the operator system 60 to be notified of the worker's physical or mental abnormality (step S408).

If neither of the two flags is set to "ON" or if the above-described series of processes has been completed for all the workers shown in the video after the processing in step S408, the process returns to step S301 in FIG. 9.

Next, we will explain the processing that occurs when the cloud system 20 notifies the provider system 60 of a malfunction prediction.

Figure 11 is a flowchart to explain the processing when a malfunction prediction is notified to the business operator system 60 from the cloud system 20 of the worker management system shown in Figure 1.

When the worker's physical and mental abnormalities are notified by the business operator communication control unit 32 of the cloud system 20 (step S501) and received by the cloud communication control unit 61 of the business operator system 60 (step S502), the business operator system 60 stores the received physical and mental abnormalities of the worker in the data storage area 62 (step S503).

The labor status management unit 63 also checks the status of workers whose physical or mental abnormalities have been reported, by telephone or in person (step S504).

Then, the results of the worker's status check are registered in the labor status management unit 63 and transmitted to the cloud system 20 under the control of the cloud communication control unit 32 (step S505).

When the worker status check result sent from the business operator system 60 is received by the business operator communication control unit 32 of the cloud system 20 (step S506), the received worker status check result is stored as business operator system production management information 31 (step S507).

If the received worker status check result indicates that the worker has no physical or mental abnormalities (step S508), the prediction accuracy is poor, and the prediction accuracy is set to "poor" (step S509).

On the other hand, if the received worker status check result indicates that the worker has a mental or physical abnormality, the prediction accuracy is set to "good" (step S510).

Here, the mental and physical disorders of workers in the cloud system 20 are judged either by using the mental and physical abnormality model 41c in the second mental and physical abnormality prediction unit 41b or by threshold judgment in the first mental and physical abnormality prediction unit 41a. Here, the target is to predict mental and physical abnormalities of workers, and since mental and physical abnormalities of workers do not occur on a daily basis, it cannot necessarily be said that sufficient learning data can be obtained early. In this embodiment, taking such circumstances into consideration, the prediction accuracy of the mental and physical abnormality prediction model is managed, and if the prediction accuracy is good, the prediction result by the mental and physical abnormality prediction model is used, and if the prediction accuracy is not good, the prediction result obtained by comparing the index value of vital information with a threshold is used.

In other words, the prediction control unit 44 sets the prediction accuracy of the mental and physical abnormality prediction model 41c based on performance information, and if the prediction accuracy is set to good, the prediction result of the second mental and physical abnormality prediction unit 41b is provided to the business operator system 60, and if the prediction accuracy is set to poor, the prediction result of the first mental and physical abnormality prediction unit 41a is provided to the business operator system 60. In this way, the prediction unit is switched based on the prediction accuracy. This makes it possible to make appropriate predictions whether or not sufficient learning data has been obtained.

In addition, in this embodiment, mental and physical abnormalities of workers are not predicted directly from images, but vital information is first calculated and mental and physical abnormalities are predicted based on that vital information. Therefore, even a relatively simple prediction method that compares an index value of vital information with a threshold value can make predictions with sufficient accuracy for practical use by utilizing existing knowledge regarding the relationship between vital information and the mental and physical state of workers.

In addition, in the cloud system 20, even while the prediction accuracy is set to poor, the second mental and physical abnormality prediction unit 41b obtains the results of the prediction of the worker's mental and physical abnormality from the output of the mental and physical abnormality prediction model 41c, and sets the quality of the prediction accuracy by comparing the performance information obtained when the first mental and physical abnormality prediction unit 41a predicts a mental and physical abnormality with the prediction results by the second mental and physical abnormality prediction unit 41b. This makes it possible to determine and set the prediction accuracy by the second mental and physical abnormality prediction unit 41b while using the prediction results by the first mental and physical abnormality prediction unit 41a.

In addition, in the cloud system 20, even while the prediction accuracy is set to poor, the second mental and physical abnormality prediction unit 41b obtains the results of prediction of the worker's mental and physical abnormality from the output of the mental and physical abnormality prediction model 41c, and the performance information obtained when the first mental and physical abnormality prediction unit 41a predicts illness is combined with the prediction results by the second mental and physical abnormality prediction unit 41b to create learning data. This allows the mental and physical abnormality prediction model 41c to be trained while using the prediction results by the first mental and physical abnormality prediction unit 41a.

Next, we will explain the process for creating the mental and physical abnormality prediction model 41c.

Figure 12 is a flowchart for explaining the process of creating the mental and physical abnormality prediction model 41c in the worker management system shown in Figure 1.

In the AI server 40 of the cloud system 20, a mental and physical abnormality prediction model 41c is created by machine learning based on the vital sign information and movement information, as well as the mental and physical abnormality occurrence history and prediction accuracy stored in step S305 of FIG. 9 (steps S601, S602). For the vital sign information and movement information, information stored as the worker's attribute information 27 is referenced, for the mental and physical abnormality occurrence history, past mental and physical abnormality occurrence history stored in the business operator system production management information 31 is referenced, and for the prediction accuracy, the prediction accuracy set in FIG. 11 is referenced.

After that, once the mental and physical abnormality prediction model 41c has been created, the process proceeds to step S301 in FIG. 9.

Next, we will explain the process for creating the work defect prediction model 42a.

Figure 13 is a flowchart for explaining the process for creating the work defect prediction model 42a in the worker management system shown in Figure 1.

In the AI server 40 of the cloud system 20, a work defect prediction model 42a is created by machine learning based on the vital signs and operation information stored in step S305 of FIG. 9, as well as the defect occurrence history, prediction accuracy, and video of the surrounding area at the time of defect occurrence (steps S701 and S702).

For vital information and movement information, information stored as worker attribute information 27 is referenced, for defect occurrence records, past defect occurrence records stored in business operator system production management information 31 are referenced, for prediction accuracy, the prediction accuracy set in FIG. 11 is referenced, and for video of the surrounding area at the time of defect occurrence, video stored in data storage area 26 is referenced. At this time, information stored as worker attribute information 27 is referenced as movement information, and this movement information is, for example, information that represents the movement by the positions of points including each of the worker's joints obtained using motion capture.

In this way, the event prediction model includes as inputs vital information and motion information represented by the positions of multiple joints of the worker, making it possible to maintain and manage good work by the worker. The event prediction model may also include as inputs an image of the worker's surroundings. For example, if an obstructive object is placed near the worker, that object will appear in the image. In this case, the predictions of the event prediction model reflect the correlation between the objects placed near the worker and the occurrence of an event.

After that, once the work defect prediction model 42a has been created, the process proceeds to step S301 in FIG. 9.

Next, we will explain the process for generating advice based on a worker's vital signs and movement information.

Figure 14 is a flowchart to explain the process of generating advice based on vital information and movement information of a worker in the worker management system shown in Figure 1.

In the AI server 40 of the cloud system 20, an advice database 48 is created by machine learning based on the vital information and operation information, as well as the defect occurrence history, prediction accuracy, and surrounding video at the time of defect occurrence, stored in step S305 of FIG. 9, in which the vital information and operation information are associated with work quality (steps S801, S802). For the vital information and operation information, information stored as the worker's attribute information 27 is referenced, for the defect occurrence history, past defect occurrence history stored in the business operator system production management information 31 is referenced, for the prediction accuracy, the prediction accuracy set in FIG. 11 is referenced, and for the surrounding video at the time of defect occurrence, the video stored in the data storage area 26 is referenced.

After that, once the advice database 48 has been created, the process proceeds to step S301 in FIG. 9.

In this way, advice is generated according to the worker's vital information and motion information, which can support more effective maintenance and management of the worker's work. For example, when a near miss is predicted during the worker's work, the worker can be notified of advice to prevent the near miss or an accident resulting from it. A near miss is an event that occurs before a serious accident occurs. Furthermore, in step S801 above, vital information and work quality can be associated by machine learning by adding information on past records of well-performed work (not shown) instead of or in addition to past records of defective occurrences. This makes it possible to notify the worker of advice for performing better work (best practices).

Next, we will explain the process of using AI to detect mental and physical abnormalities in workers.

Figure 15 is a flowchart to explain the process of detecting mental and physical abnormalities of workers using AI in the worker management system shown in Figure 1.

In the labor management control unit 28 of the cloud system 20, the occurrence of mental and physical abnormalities of workers is predicted based on the vital information and movement information stored in step S305 of FIG. 9, using work process information linked to the existing system and the mental and physical abnormality prediction model 41c (step S901).

If the occurrence of a mental or physical abnormality in the worker is predicted (step S902), the operator system 60 is notified of the prediction of a mental or physical abnormality in the worker under the control of the operator communication control unit 32 (step S903).

Then, after the operator system 60 is notified by the control of the operator communication control unit 32 that a mental or physical abnormality of the worker has been predicted, or if no mental or physical abnormality of the worker is predicted, the process proceeds to step S301 in FIG. 9.

Next, we will explain the process of using AI to predict work defects and mistakes.

Figure 16 is a flowchart to explain the process of predicting work defects and errors using AI in the worker management system shown in Figure 1.

In the labor management control unit 28 of the cloud system 20, the occurrence of work defects is predicted based on the vital sign information and video stored in step S305 of FIG. 9, using the work process information linked to the existing system and the work defect model 42a (step S1001).

If a work defect is predicted (step S1002), the operator system 60 is notified by the control of the operator communication control unit 32 that a work defect by the worker is predicted (step S1003).

Then, after the operator system 60 is notified by the control of the operator communication control unit 32 that a work defect by the worker is predicted, or if a work defect by the worker is not predicted, the process proceeds to step S301 in FIG. 9.

In this way, in addition to predicting mental and physical abnormalities of workers, defects in the work of workers can also be predicted, which can support more effective maintenance and management of the work of workers.

Next, we will explain the process of checking workers' vital signs and other information in the business operator system 60.

Figure 17 is a flowchart to explain the process of checking vital signs and other information of workers in the operator system 60 of the worker management system shown in Figure 1.

When checking a worker's vital signs and other information in the operator system 60 of the worker management system shown in FIG. 1, the cloud system 20 is accessed under the control of the cloud communication control unit 61 of the operator system 60, and the labor status management unit 63 requests the collection of vital signs and images (step S1101). When this request is received by the operator communication control unit 32 of the cloud system 20 (step S1102), the labor management control unit 28 of the cloud system 20 obtains the vital signs and images, as well as advice data if available (step 1103), and these data are sent to the operator system 60 under the control of the operator communication control unit 32 (step S1104).

When the vital information and images sent from the cloud system 20 are received by the cloud communication control unit 61 of the business operator system 60 (step S1105), these data are stored in the data storage area 62 (step S1106).

The labor status management unit 63 also displays the vital signs information and images sent from the cloud system 20 (step S1107).

Next, we will explain the process of pushing work-related precautions and other information to workers.

Figure 18 is a flowchart to explain the process of pushing work-related precautions and other information to workers in the worker management system shown in Figure 1.

In the labor management control unit 28 of the cloud system 20, advice and best practice videos that are to be considered as attitudes and actions for the corresponding process are extracted from the vital information and videos saved in step S305 of FIG. 9 (step S1201). This is done by referring to the worker attribute information 27, work process information linked to the existing system, the work defect prediction model 42a, and the advice database 48.

Once the advice and best practice video have been extracted, this information is sent to the mobile terminal 50 under the control of the worker communication control unit 30 (step S1202).

After the advice and best practice video sent from the cloud system 20 are received by the cloud communication control unit 51 of the mobile terminal 50 (step S1203), the received advice and best practice video are displayed on the information display unit 52 (step S1204).

Next, we will explain the process by which workers check vital signs trends using their own mobile terminals 50.

Figure 19 is a flowchart to explain the process by which a worker checks vital signs trends and the like using his or her own mobile terminal 50 in the worker management system shown in Figure 1.

When a worker checks his/her own vital signs in the worker management system shown in FIG. 1, the cloud system 20 is accessed using the mobile terminal 50 under the control of the cloud communication control unit 51 of the mobile terminal 50, and a request is made to check the trends of the vital signs (step S1301). When this request is received by the worker communication control unit 30 of the cloud system 20 (step S1302), the information provision control unit 29 of the cloud system 20 acquires the vital signs and images, as well as advice data if available (step 1303), and trend information showing the worker's tendencies as indicated by these data is transmitted to the mobile terminal 5 under the control of the worker communication control unit 30 (step S1304).

After the trend information indicating the worker's tendencies transmitted from the cloud system 20 is received by the cloud communication control unit 51 of the mobile terminal 50 (step S1305), the received trend information is displayed on the information display unit 52 (step S1306).

In this manner, in this embodiment, the cloud system 20 receives a request from the mobile terminal 50, generates trend information on the work of the worker based on the correlation between the worker's past vital information and the work quality, and transmits the trend information indicating the trend to the mobile terminal 50. This makes it possible to support more effective maintenance and management of the work performed by the worker.

A trend in the work of a worker is, for example, information that indicates that there is a high possibility of demonstrating best practices when the worker's vital signs meet certain conditions. Or, for example, information that indicates that there is a high possibility of a near miss occurring when the worker's vital signs meet certain conditions.

As described above, in this embodiment, the occurrence of a specific event related to a worker is predicted using an event prediction model, the worker is confirmed, and the confirmation result is reflected as learning data for the event prediction model, making it possible to maintain and manage good work by the worker.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and these embodiments may be used in combination or some of the configuration may be changed within the scope of the technical concept of the present invention.

10...Video recording device, 11...Camera, 12, 22...CPU, 13, 23...Memory, 14, 24...Facial recognition control unit, 15...Video vitals control unit, 16, 25...Behavioral analysis control unit, 17, 51, 61...Cloud communication control unit, 18...Camera image control unit, 19, 26, 62...Data storage area, 20...Cloud system, 21...Video recording device communication control unit, 27...Worker attribute information, 28...Labor management control unit, 29...Information provision control unit, 30...Worker communication control unit, 31...Business operator system System production management information, 32...Business operator communication control unit, 40...AI server, 41...Mental and physical abnormality prediction unit, 41a...First mental and physical abnormality prediction unit, 42a...Second mental and physical abnormality prediction unit, 41c...Mental and physical abnormality prediction model, 42...Work defect prediction unit, 42a...Work defect prediction model, 43...Work status prediction unit, 44...Prediction control unit, 45, 46...Learning unit, 47...Advice unit, 48...Database, 50...Mobile terminal, 52...Information display unit, 60...Business operator system, 63...Labor status management unit, 64...Business management system

Claims (4)

A camera for taking video of workers engaged in work;
a processing device that predicts the occurrence of a predetermined event related to the worker based on vital information of the worker obtained from the captured video and motion information represented by positions of a plurality of joints of the worker ;
a management device that indicates that the occurrence of the predetermined event is predicted,
The processing device learns the learning data to generate a mental and physical abnormality prediction model that receives the worker's vital information and motion information as input and outputs a prediction result of the occurrence of mental and physical abnormalities related to the worker, and predicts the occurrence of mental and physical abnormalities related to the worker using the mental and physical abnormality prediction model;
the management device acquires performance information indicating whether the predicted mental and physical abnormality has occurred from the worker, and provides the performance information to the processing device as data to be added to the learning data;
The processing device includes:
a first mental and physical abnormality prediction unit that predicts the occurrence of a mental and physical abnormality of the worker based on a result of comparing an index value of the vital information with a predetermined threshold value and the motion information without using the mental and physical abnormality prediction model;
a second mental and physical abnormality prediction unit that inputs the vital information and the motion information into the mental and physical abnormality prediction model and obtains a prediction result of the occurrence of a mental and physical abnormality of the worker from an output of the mental and physical abnormality prediction model;
a prediction control unit that compares the performance information with a result of prediction by a second mental and physical abnormality prediction unit to determine whether the prediction accuracy by the mental and physical abnormality prediction model is good or bad, and if the prediction accuracy is good, provides the prediction result by the second mental and physical abnormality prediction unit to the management device, and if the prediction accuracy is not good, provides the prediction result by the first mental and physical abnormality prediction unit to the management device;
having
Even while the prediction accuracy is not good, the second mental and physical abnormality prediction unit obtains a prediction result of the mental and physical abnormality of the worker from the output of the mental and physical abnormality prediction model, and determines whether the prediction accuracy is good or bad by comparing performance information obtained when a mental and physical abnormality is predicted by the first mental and physical abnormality prediction unit with the prediction result by the second mental and physical abnormality prediction unit, and combines the performance information with the vital information and the motion information to form the learning data.
Worker management support system. The processing device further includes a work defect prediction unit that inputs vital information and motion information of a worker into a work defect prediction model that inputs the vital information and motion information of a worker and outputs a prediction result of the occurrence of work defects by the worker, and obtains a prediction result of the occurrence of work defects by the worker from the output of the work defect prediction model.
2. The worker management support system according to claim 1 . The processing device is realized on a cloud.
2. The worker management support system according to claim 1. A computer system for predicting the occurrence of a predetermined event related to a worker based on vital information of the worker obtained from a video of the worker engaged in work captured by a camera device and motion information represented by positions of a plurality of joints of the worker, and presenting a message indicating that the occurrence of the predetermined event has been predicted, comprising:
By learning the learning data, a mental and physical abnormality prediction model is generated, which inputs the vital information and movement information of the worker and outputs a prediction result of the occurrence of mental and physical abnormalities related to the worker, and predicts the occurrence of mental and physical abnormalities related to the worker using the mental and physical abnormality prediction model;
acquiring performance information from the worker indicating whether the predicted mental and physical abnormality has occurred, and providing the performance information as data to be added to the learning data,
a first prediction process for predicting the occurrence of a mental or physical abnormality of the worker based on a result of comparing an index value of the vital information with a predetermined threshold value and the motion information without using the mental or physical abnormality prediction model;
a second prediction process for inputting the vital information and the movement information into the mental and physical abnormality prediction model and obtaining a prediction result of the occurrence of a mental and physical abnormality of the worker from an output of the mental and physical abnormality prediction model;
The performance information is compared with the result of the prediction by the second prediction process to determine whether the prediction accuracy by the mental and physical abnormality prediction model is good or bad, and if the prediction accuracy is good, the prediction result by the second prediction process is provided, and if the prediction accuracy is not good, the prediction result by the first prediction process is provided;
Even when the prediction accuracy is not good, a prediction result of the mental and physical abnormality of the worker is obtained from the output of the mental and physical abnormality prediction model as the second prediction process, and performance information obtained when a mental and physical abnormality is predicted in the first prediction process is compared with the prediction result by the second prediction process to determine whether the prediction accuracy is good or bad, and the performance information is combined with the vital information and the motion information to form the learning data.
Methods for supporting worker management.

Images