CN118736683A - A method and system for monitoring operator violations based on a control ball - Google Patents

Abstract

The present invention relates to the field of image processing, and the present invention relates to a method and system for monitoring violations of operating personnel based on a control ball, the method comprising: framing a video; calculating the importance of a target frame image, the importance being positively correlated with the action significance of key points of a human skeleton in the target frame image; calculating the sum of the differences in importance between a target frame image and other frame images in a second sequence and normalizing them, and when the sum of the normalized differences is less than or equal to a set threshold, dividing the target frame image into key frames; traversing to obtain all key frames; inputting the key points in all key frames into a trained spatiotemporal graph convolutional network, and outputting the classification results of human actions in the key frames. The present invention processes by framing and calculating the importance of each frame, thereby screening out key frames, and integrating deep learning technology to monitor violations, thereby reducing the number of frames that need to be reviewed and improving the efficiency and accuracy of monitoring.

Description

A method and system for monitoring operator violations based on a control ball

Technical Field

The present invention relates to the field of image processing and a method and system for monitoring violations of operating personnel based on a control ball.

Background Art

With the rapid development of the construction industry, the safety protection of workers has become a top priority. However, since most workers have a weak awareness of safety protection during construction, and the potential safety hazards during construction are often not easily noticed, if workers fail to comply with the safety regulations of the construction site, such as fighting each other, illegal climbing, and damaging public property, the consequences are unpredictable.

As a highly integrated monitoring product, the control ball plays a vital role in the construction industry. The control ball system can integrate intelligent analysis algorithms to automatically identify violations by operators, such as not wearing a safety helmet, illegal climbing, fighting, unauthorized operation of equipment, etc., and immediately trigger an alarm so that management personnel can intervene in time.

The patent application document with publication number CN117253329A discloses a method and system for monitoring and warning of personnel violations in the production area of a power plant. The method includes: installing surveillance cameras to sample data on personnel activities; using video analysis technology to perform real-time analysis of human posture and movements; the real-time analysis of human posture and movements using video analysis technology is achieved through a recurrent neural network, which constructs the action and behavior information of the input sequence through time loops and hidden state transfers, and captures the temporal dependencies, thereby realizing the recognition and analysis of actions; judging personnel violations according to violation rules; and notifying the violation results.

However, the use of the above-mentioned video analysis technology requires a large amount of computing resources, especially in high-resolution and high-frame-rate video processing. At the same time, a large amount of data needs to be acquired and labeled before model training, which is not only time-consuming and labor-intensive, but also requires powerful computing equipment support.

Summary of the invention

In order to solve the above technical problems that a large amount of computing resources are required and data preparation is time-consuming and labor-intensive, the present invention provides solutions in the following aspects.

In a first aspect, a method for monitoring operator violations based on a control ball comprises:

The video shot by the control ball is divided into frames and pre-processed to obtain each frame image;

Calculating the importance of a target frame image, the target frame image being any frame image; the importance being positively correlated with the motion significance of key points of a human skeleton in the target frame image;

Taking the target frame image as the center, intercepting several adjacent frame images to construct a second sequence, calculating and normalizing the sum of the importance differences between the target frame image and other frame images in the second sequence, and when the sum of the normalized differences is less than or equal to the set threshold, the target frame image is divided into a key frame; traversing to obtain all key frames;

The key points in all key frames are input into the trained spatiotemporal graph convolutional network, and the classification results of human actions in the key frames are output.

Beneficial effects: The present invention first divides the video into frames and pre-processes it, and then evaluates the importance of the image of the corresponding frame by calculating the motion significance of the key points of the human skeleton in each frame, thereby effectively screening out key frames with potential violations from a large amount of continuous video data, reducing missed detections and false alarms, and integrating deep learning technology to monitor violations, thereby reducing the number of frames that need to be reviewed and improving monitoring efficiency and accuracy.

Preferably, the importance satisfies the relationship:

; In the formula, is the importance of the target frame image, is the first The action significance of the key points, For the The weight corresponding to the key point is is the number of key points in the target frame image, Indicates normalization processing.

Beneficial effects: Taking into account the motion significance of each key point, it ensures that the contribution of the motion of each key point in the image to the overall importance can be fully reflected, which helps to capture small but critical motion changes, thereby improving the recognition rate of violations.

Preferably, the importance is the maximum value of the product of the action significance of all key points and their corresponding weights.

Beneficial effect: Weights are set for different key points, and the maximum weighted action significance is selected as the measure of importance. Since the focus is on the most significant actions, even short-term illegal actions are difficult to escape the monitoring of the system, thus reducing the possibility of missed detection.

Preferably, the process of acquiring the action saliency includes:

Calculate the The trajectory change rate and relative change rate of the key points are weighted and summed to obtain the action significance;

Among them, taking the target frame image as the center, intercepting a certain frame of images to construct the first sequence, and calculating the The key points in the target frame image are respectively The average value of the sum of the rates of the key points in other frame images is used as the trajectory change rate;

Among them, calculate the The first average distance between the key point and all other key points in the target frame image is calculated. The second average distances of the key points in other frame images in the first sequence are calculated, and the variance of the first average distance and all the second average distances is calculated, and the variance is used as the relative change rate.

Beneficial Effects: Calculating the mean of the sum of velocities and the variance of the mean distance helps capture even small changes in motion and increases sensitivity to abnormal behavior.

Preferably, the process of acquiring the action saliency includes:

Calculate the motion vector of the key point between consecutive frames using an optical flow method, wherein the motion vector includes a vector magnitude;

The variance of the motion vector magnitude is calculated, and the variance is used as the motion saliency.

Preferably, an optical flow method is used to calculate a motion vector of a key point between consecutive frames, wherein the motion vector includes a vector magnitude; an average value of the motion vector magnitudes is calculated, and the average value of the motion vector magnitudes is used as the action saliency.

Beneficial effects: The optical flow method can accurately capture the dynamic changes of key points between consecutive frames. Whether calculating the variance or the average value, it can easily adapt to different monitoring environments and needs.

Preferably, the spatiotemporal graph convolutional network uses a cross entropy loss function during training.

In a second aspect, a system for monitoring violations of operating personnel based on a control ball is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, any one of the methods for monitoring violations of operating personnel based on a control ball is implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

FIG. 1 is a method flow chart of steps S1 to S4 in a method for monitoring operator violations based on a control ball according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

1 , a method for monitoring operator violations based on a control ball includes steps S1 to S4, which are as follows:

S1: Frame and pre-process the video shot by the control ball to obtain each frame image.

The control ball can monitor the behavior and posture of workers in real time, and promptly detect potential violations or unsafe actions.

In an embodiment of the present invention, a control ball is installed in the working environment according to actual conditions, and can capture a video of the posture of the workers. By decomposing the video into separate frames, key frames can be analyzed more specifically instead of processing the entire video sequence, thereby improving data processing efficiency.

Since the control ball takes a long time to work and the working environment is relatively complex, the framed image may contain noise and other factors that affect the quality. Therefore, each frame image needs to be preprocessed.

Specifically, the video file collected by the control ball is input into OpenCV (Open Source Computer Vision Library), and the video is divided into frames to obtain each frame image;

Use filters such as Gaussian filtering to remove noise from each frame of the image, and use contrast enhancement algorithms to improve the quality of the image.

S2: Calculate the importance of a target frame image, where the target frame image is any frame image; the importance is positively correlated with the motion significance of key points of a human skeleton in the target frame image.

In video analysis, key points usually refer to prominent and characteristic points in an image, which can be tracked between images to analyze the motion, posture or other attributes of an object. For the human body, key points generally refer to the joint positions of the human body, such as the head, shoulders, elbows, wrists, hips, knees and ankles. When analyzing, not only the movement of individual key points should be considered, but also the relative movement between key points.

COCO (Common Objects in Context) dataset The COCO dataset contains 91 types of targets, covering more than 80 categories. These categories include people, various animals, vehicles, furniture, etc., which are not only used for target detection, but also involve tasks such as image description generation and human pose estimation. It has very high diversity and application value. Specifically, the Pose Estimation model in the COCO (Common Objects in Context) dataset is first used to process each frame of the image to detect the key points of the human body in the image.

In the embodiment of the present invention, a rectangular coordinate system is established with the pixel point at the lower left corner of each frame of image as the coordinate origin, with the x-axis pointing horizontally to the right and the y-axis pointing vertically upward.

In order to capture the details of the action changes, any frame image is selected as the target frame image, and a certain frame of images is captured with the target frame image as the center to construct the first sequence. For example, with the target frame image as the center, one frame of images is captured forward and backward in chronological order to construct the first sequence.

For example, the frame rate of the captured video is set to , then the time interval between each frame is , and the key points of the human body in each frame are complete and unobstructed. The target frame image is taken as an example, and the other two frames in the first sequence are marked as Frame image, Frame image, obtain the coordinates of all key points in the target frame image, among which, select the first For example, the key point The coordinates of the key points are , similarly, get The corresponding coordinates of the key points in the other two frames are marked as , .

Then calculate the The trajectory change rate of the key points satisfies the following relationship:

In the formula, For the Frame target frame image The trajectory change rate of key points, Indicates The first The key points and The first frame in the target frame image The distance between key points, Indicates The first The key points and The first frame in the target frame image The distance between key points.

in, It reflects the sum of the rates of the same key point between three consecutive frames.

It should be noted that in calculating the After calculating the trajectory change rate of each key point, normalization can be performed.

In addition, calculate the first The Euclidean distance between the key point and other key points is calculated, and the sum of all the calculated Euclidean distances is taken as the first average distance. Similarly, the calculations are respectively The second average distance of the key point in the other two frames of images, calculate the variance of the first average distance and the two second average distances, and use the calculated variance as the first The relative rate of change of a key point.

The trajectory change rate and relative change rate calculated above are weighted and summed to obtain the first The action significance of the key points satisfies the following relationship:

In the formula, Indicated in Frame target frame image The action significance of the key points, Indicated in Frame target frame image The trajectory change rate of key points, Indicated in Frame target frame image The relative change rate of the key points, , All are weights.

In another embodiment, in the Frame target frame image The action salience of key points It also satisfies the following relationship:

In the formula, Indicated in Frame target frame image The action significance of the key points, Indicated in Frame target frame image The trajectory change rate of key points, Indicated in Frame target frame image The relative change rate of the key points, represents normalization processing, It is a logistic function that maps the input to a value between 0 and 1. The larger the input value, the closer the output value is to 1, indicating that the action significance is higher.

In another embodiment, the optical flow method can be used to calculate the The motion vector of the key point between consecutive frames, including the magnitude and direction, calculates the average or variance of the motion vector magnitude and uses it as the first The larger the mean or variance, the more significant the motion.

According to the above calculation The action significance of the key point can be calculated in the same way. The action significance of all other key points in the target frame image can be calculated. The importance of the target frame image satisfies the following relationship:

; In the formula, For the The importance of the frame target frame image, For the Frame target frame image The action significance of the key points, For the The weight corresponding to the key point is is the number of key points in the target frame image, represents the normalization process. Among them, the action saliency The bigger, The higher the value, the more likely the target frame image is to contain important actions or events.

It should be noted that Can be set based on experience.

In another embodiment, the maximum value of the product of the action significance of all key points and their corresponding weights may be used as the importance of the target frame image.

S3: Taking the target frame image as the center, intercepting several adjacent frame images to construct a second sequence, calculating the sum of the differences in importance between the target frame image and other frame images in the second sequence and normalizing them, and when the sum of the normalized differences is less than or equal to the set threshold, dividing the target frame image into a key frame; traversing to obtain all key frames.

In the embodiment of the present invention, the first The importance of the target frame image can be calculated, and then the importance of all frame images can be calculated.

Specifically, the The target frame image is taken as the center, and several adjacent frame images are intercepted to construct the second sequence. The sum of the differences between the importance of the target frame image and all other frame images is normalized. When the sum of the normalized differences is less than or equal to the set threshold, the first The target frame image is divided into key frames; all key frames can be obtained in the same way.

It should be noted that the threshold here can be set to 0.6 based on experience.

S4: Input the key points in all key frames into the trained spatiotemporal graph convolutional network and output the classification results of human actions in the key frames.

In the embodiment of the present invention, the key points in all key frames obtained in the above step S3 are input into the trained spatiotemporal graph convolutional network (ST-GCN), and the classification results of human actions in the key frames are output, which is an efficient action recognition method. The general steps are as follows:

Use a pose estimation algorithm (such as the one in the COCO library) to detect the key points of the human body in each keyframe;

The key points and their relationships are constructed into a graph structure, where each key point is a node and the relationship between nodes defines the edges of the graph;

Extract the features of each key point, such as position coordinates, velocity, acceleration, etc.;

Use the annotated action data to train the ST-GCN network and learn the spatiotemporal features of key points, where the annotations include key point coordinates, key point trajectories, action categories, time information, etc.

The key point data in the extracted key frame is input into the trained ST-GCN network. The network learns the spatial relationship between key points through the graph convolution layer and the change of key points over time through the temporal convolution layer.

The network outputs the action classification result for each keyframe.

Training a spatiotemporal graph convolutional network is an existing technology and will not be described in detail here. It should be noted that when training a spatiotemporal graph convolutional network, the cross entropy loss function is preferably used.

The present invention first divides the video into frames and pre-processes it, and then evaluates the importance of the image of the corresponding frame by calculating the motion significance of the key points of the human skeleton in each frame of the image, thereby effectively screening out key frames with potential violations from a large amount of continuous video data, reducing missed detections and false alarms, and integrating deep learning technology to monitor violations, thereby reducing the number of frames that need to be reviewed and improving the efficiency and accuracy of monitoring.

The system includes a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, the method for monitoring operator violations based on a control ball according to the first aspect of the present invention is implemented.

The system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

In the present invention, the aforementioned memory may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module or both. Any such computer storage medium may be part of a device or accessible or connectable to a device.

In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims (8)

1. The utility model provides a method for monitoring operators' violations based on control ball, which is characterized by comprising:

framing and preprocessing videos shot by the control ball to obtain each frame of image;

Calculating the importance degree of a target frame image, wherein the target frame image is any frame image; the importance degree is positively related to the action significance of key points of the human skeleton in the target frame image;

Taking the target frame image as a center, intercepting a plurality of adjacent frame images to construct a second sequence, calculating the sum of differences of importance degrees of the target frame image and other frame images in the second sequence, normalizing, and dividing the target frame image into key frames when the sum of the normalized differences is smaller than or equal to a set threshold value; traversing to obtain all key frames;

and inputting the key points in all the key frames into a trained space-time diagram convolution network, and outputting classification results of human actions in the key frames.

2. The method for monitoring the violation of a worker based on a control ball according to claim 1, wherein the importance degree satisfies the relation:

; in the method, in the process of the invention, For the importance level of the target frame image,Is the first in the target frame imageThe significance of the action of the individual key points,Is the firstThe weights corresponding to the individual key points are used,For the number of keypoints in the target frame image,The normalization process is represented.

3. The method for monitoring operator violations based on the control balls according to claim 1, wherein the importance degree is a maximum value of products of action salience of all key points and weights corresponding to the key points.

4. The method for monitoring the violation of the operator based on the control ball according to claim 2, wherein the process for obtaining the action significance comprises the following steps:

Calculate the first The track change rate and the relative change rate of each key point are weighted and summed to obtain the action significance;

Taking the target frame image as the center, intercepting the image of a certain frame to construct a first sequence, and calculating the first sequence The key points are respectively matched with the first in the target frame imageAn average value of the sum of rates of the key points in other frame images, wherein the average value is used as the track change rate;

Wherein, calculate the first Calculating the first average distance between each key point and all other key points in the target frame imageAnd calculating the variance of the first average distance and all the second average distances of the key points in the second average distances of other frame images in the first sequence, and taking the variance as the relative change rate.

5. The method for monitoring the violation of an operator based on a control ball according to claim 3, wherein the process for obtaining the action significance comprises the following steps:

Calculating motion vectors of key points between successive frames using optical streaming, the motion vectors comprising vector magnitudes;

And calculating the variance of the motion vector amplitude, and taking the variance as the motion significance.

6. A method of monitoring operator violations based on a control sphere according to claim 3, characterized in that the motion vectors of key points between successive frames are calculated using an optical flow method, said motion vectors comprising vector magnitudes; and calculating an average value of the motion vector amplitude, and taking the average value of the motion vector amplitude as the motion significance.

7. The method for monitoring operator violations based on a control sphere according to claim 1, wherein said space-time diagram convolution network uses a cross entropy loss function in the training process.

8. An operation personnel monitoring system that violating regulations based on cloth accuse ball, characterized by comprising: a processor and a memory storing computer program instructions that when executed by the processor implement the method of cloth ball based operator violation monitoring of any of claims 1-7.