CN103716579A - Video monitoring method and system - Google Patents

Abstract

A video monitoring method: collecting image frames of multiple cameras, extracting moving foreground according to the image frames; obtaining pixel coordinates of the moving foreground in the image frames; according to the pixel coordinates, the focal lengths and The spatial distance is used to calculate the three-dimensional space coordinates of the moving foreground; an abnormal event of the moving foreground is triggered according to the three-dimensional space coordinates. In addition, a video monitoring system is also provided. The above video monitoring method and system can improve the accuracy of abnormal event triggering, thereby improving security.

Description

Video surveillance method and system

technical field

The invention relates to the technical field of image processing, in particular to a video monitoring method and system.

Background technique

In the traditional video monitoring method, a camera is usually set in advance at the monitoring site, and the video data of the monitoring site is collected through the camera, and the video data is usually in the form of continuous image frames. The image frame includes a background and a moving foreground, and the moving foreground is an area where pixels in the image frame change. For example, when a pedestrian passes by the monitoring scene, the moving area occupied by the pedestrian in the acquired continuous image frames is the moving foreground, while the relatively static monitoring scene is the background part of the image frame.

In the traditional technology, the moving foreground of the image frame is usually extracted first, and then it is detected whether the extracted moving foreground enters a pre-defined dangerous area, so as to judge whether there is an abnormal situation and issue a corresponding alarm.

However, in the video surveillance method in traditional technology, the image frame obtained by the camera can only reflect the plane information of the surveillance scene. When the moving foreground moves axially in the focal length direction of the camera, it is impossible to judge whether it enters the dangerous area, which makes the monitoring process difficult. Omit some abnormal conditions and send out alarms, thereby reducing safety.

Contents of the invention

Based on this, it is necessary to provide a video monitoring method that can improve security.

A video surveillance method, comprising:

Collect image frames of multiple cameras, and extract moving foreground according to the image frames;

Obtaining the pixel coordinates of the moving foreground in the image frame;

calculating the three-dimensional spatial coordinates of the moving foreground according to the pixel coordinates, focal lengths and spatial distances of multiple cameras;

An abnormal event of the moving foreground is triggered according to the three-dimensional space coordinates.

In one of the embodiments, the step of extracting the moving foreground according to the image frame is:

The moving foreground of the image frame is extracted by background difference according to the mixed Gaussian model.

In one of the embodiments, the number of the cameras is 2, and they are arranged horizontally;

The acquired image frames are left image frames and right image frames collected respectively by the two cameras at the same time;

The step of calculating the three-dimensional spatial coordinates of the moving foreground according to the pixel coordinates, the focal length and the spatial distance of multiple cameras is:

According to the formula:

Disparity＝X _left -X _right

Calculate the observation of the left image frame and the right image frame, wherein Disparity is a parallax, X _left is the horizontal coordinate of the moving foreground in the left image frame, and X _right is the horizontal coordinate of the moving foreground in the right image frame ;

According to the formula:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; BX</span>}}_{\mathrm{<span\; class="notranslate">\; left</span>}}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; BY</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\end{array}\right.$

Calculate the three-dimensional space coordinates of the moving foreground; where (x _c , y _c , z _c ) are the three-dimensional space coordinates of the moving foreground, and x _c and y _c are visual plane coordinates, and z _c is depth information , B is the horizontal distance between the two cameras, f is the focal length of the camera, Y is the vertical coordinate of the moving foreground in the left image frame and the right image frame, Disparity is the left image frame and the right image frame The disparity of the image frame.

In one of the embodiments, the step of triggering the abnormal event of the moving foreground according to the three-dimensional space coordinates is:

Judging whether the three-dimensional space coordinates are located in a preset dangerous area, and if so, triggering an abnormal event of the moving foreground.

In one of the embodiments, the step of judging whether the three-dimensional space coordinates are located in a preset dangerous area is as follows:

Acquiring the depth image corresponding to the image frame, the depth image is a gray scale image, the coordinates of its pixel points correspond to the visual plane coordinates of the three-dimensional space coordinates, and its gray value corresponds to the depth information of the three-dimensional space coordinates;

Whether the three-dimensional space coordinates are located in a preset dangerous area is judged through the depth image.

In one of the embodiments, before the step of triggering the abnormal event of the moving foreground, the step further includes:

Judging whether the moving foreground is a human body gesture image according to the depth information, and if so, continue to execute the step of triggering the abnormal event of the moving foreground.

In one of the embodiments, the step of judging whether the moving foreground is a human body gesture image according to the depth information is:

According to the formula:

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; u</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Calculate the depth value of the moving foreground, wherein d (I, c) is the depth value of the pixel at coordinate c in the depth image I, u and v are the coordinates of any two points randomly selected within the threshold range, and z is Depth information on the motion foreground;

Obtain a training classifier for preset human gesture images;

According to the depth value of the moving foreground and the training classifier, it is judged whether the moving foreground is a human gesture image through a decision tree.

In addition, it is also necessary to provide a video surveillance system that can improve security.

A video surveillance system comprising:

Multi-eye camera for collecting image frames;

A moving foreground extracting device, configured to extract a moving foreground according to the image frame;

A coordinate conversion device, configured to obtain the pixel coordinates of the moving foreground in the image frame, and calculate the three-dimensional space coordinates of the moving foreground according to the pixel coordinates, the focal length and the spatial distance of the multi-eye camera;

An abnormal event triggering device, configured to trigger an abnormal event of the moving foreground according to the three-dimensional space coordinates.

In one of the embodiments, the moving foreground extracting device is further configured to extract the moving foreground of the image frame by background difference according to the mixed Gaussian model.

In one of the embodiments, the multi-eye camera is a horizontally arranged binocular camera;

The acquired image frame is a left image frame and a right image frame collected by the binocular camera at the same time;

The coordinate conversion device is also used for according to the formula:

Disparity＝X _left -X _right

Calculate the observation of the left image frame and the right image frame, wherein Disparity is a parallax, X _left is the horizontal coordinate of the moving foreground in the left image frame, and X _right is the horizontal coordinate of the moving foreground in the right image frame , and according to the formula:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; BX</span>}}_{\mathrm{<span\; class="notranslate">\; left</span>}}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; BY</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\end{array}\right.$

Calculate the three-dimensional space coordinates of the moving foreground; where (x _c , y _c , z _c ) are the three-dimensional space coordinates of the moving foreground, and x _c and y _c are visual plane coordinates, and z _c is depth information , B is the horizontal distance between the two cameras, f is the focal length of the camera, Y is the vertical coordinate of the moving foreground in the left image frame and the right image frame, Disparity is the left image frame and the right image frame The disparity of the image frame.

In one of the embodiments, the abnormal event triggering device is further used to judge whether the three-dimensional space coordinates are located in a preset dangerous area, and if so, trigger the abnormal event of the moving foreground.

In one of the embodiments, the abnormal event triggering device is also used for

Obtain the depth image corresponding to the image frame, the depth image is a grayscale image, the coordinates of its pixels correspond to the visual plane coordinates of the three-dimensional space coordinates, and its grayscale value corresponds to the depth information of the three-dimensional space coordinates, and through the The depth image judges whether the three-dimensional space coordinates are located in a preset dangerous area.

In one of the embodiments, before the step of triggering the abnormal event of the moving foreground, the step further includes:

Judging whether the moving foreground is a human body gesture image according to the depth information, and if so, continue to execute the step of triggering the abnormal event of the moving foreground.

In one of the embodiments, the abnormal event triggering device is also used according to the formula:

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; u</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Calculate the depth value of the moving foreground, wherein d (I, c) is the depth value of the pixel at coordinate c in the depth image I, u and v are the coordinates of any two points randomly selected within the threshold range, and z is Depth information on the moving foreground; obtaining a training classifier of a preset human gesture image; judging whether the moving foreground is a human gesture image through a decision tree according to the depth value of the moving foreground and the training classifier.

The above video surveillance method and device collects image frames through multi-eye cameras, extracts the moving foreground according to the image frames, then converts the pixel coordinates of the moving foreground in the image frames into the three-dimensional space coordinates of its actual position, and according to the motion The three-dimensional space coordinates of the foreground trigger abnormal events. Compared with the traditional technology, not only the abnormal events can be triggered according to the position of the moving foreground on the two-dimensional plane, but also the depth information can be obtained from the converted three-dimensional space coordinates of the moving foreground ( The distance from the camera) triggers abnormal events, which makes the triggering of abnormal events more accurate, thereby improving security.

Description of drawings

Fig. 1 is the flowchart of video monitoring method in an embodiment;

Fig. 2 is a schematic diagram of coordinate conversion of two camera imaging in one embodiment;

Fig. 3 is a schematic diagram of a depth image in an embodiment;

Fig. 4 is the structural representation of video monitoring system in an embodiment;

Fig. 5 is a schematic structural diagram of a binocular camera in an embodiment.

Detailed ways

In one embodiment, as shown in Figure 1, a method for obtaining picture information includes the following steps:

Step S102, collecting image frames of multiple cameras, and extracting a moving foreground according to the image frames.

The image data collected by the camera is in the form of continuous image frames. A moving foreground is a region of pixels that changes in successive image frames.

In one embodiment, the moving foreground of the image frame can be extracted by background subtraction according to the mixed Gaussian model.

In this embodiment, the mixed Gaussian model can be defined in advance, and the defined mixed Gaussian model is:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

$\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\frac{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span>}^{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ;</span>$

${\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; ;</span>$

为第k个高斯分布的方差，E为高斯分布的期望值。Among them, X _t is the pixel value of the pixel point (x, y) at time t (since the sampling frequency of the sampling image frame is a fixed value when the camera is shooting video images, t can also be the number of frames), and K is the preset The number of mixed Gaussian models (usually 3 to 5), w _{k, t} is the weight of the kth Gaussian distribution at the preset time t, μ _{k, t} is the mean value of the kth Gaussian distribution at time t , ∑ _{k, t} is the covariance matrix of the kth Gaussian distribution at time t, N is the probability density function of Gaussian distribution,

is the variance of the kth Gaussian distribution, and E is the expected value of the Gaussian distribution.

After the image frame is acquired, the Gaussian model can be updated according to the current moment. The online K-means algorithm can be used to approximate the parameters. The online K-means algorithm matches the pixel at the current moment with K Gaussian distributions. If the matching is successful, the mean and variance of the distribution are updated, and the weight of the distribution is increased; if there is no match, a new distribution is generated to replace it. The term with the smaller weight in the existing mixture distribution.

The aforementioned K Gaussian distributions can be arranged according to the ratio of weight and variance from large to small, and then the Gaussian distribution closest to the mean value of the distribution can be selected as the matching Gaussian distribution, namely:

${\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&lambda;\&sigma;</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; otherwise</span>}\end{array}\right.$

Among them, M _{k, t} is the matching coefficient, that is, the k value when M _{k, t} = 1 is the k value of the finally selected matching Gaussian model, X _t is the pixel value of the pixel point (x, y) at time t, μ _k,t is the mean value of the kth Gaussian distribution at time t, σ is the variance, and λ is the preset coefficient.

If no matching distribution is found for the current pixel among the K distributions, the least likely distribution is replaced by a new distribution. The mean of the new distribution is set to the current pixel value, with larger variance and smaller weight.

Then, according to the selected Gaussian model, the weights of K distributions at time t can be adjusted, and the formula can be used:

w _k,t+1 ＝(1-α)w _k,t +αM _k,t

w _k,t+1 =w _k,t +α(M _k,t -w _k,t )

Adjust the weights. Where α is not a preset coefficient, M _k,t is the aforementioned matching coefficient, w _k,t is the weight of the kth Gaussian distribution at time t. After adjusting the weights, the sum of the weights remains the same and is still 1. The update process is equivalent to performing a causal second-order low-pass filter on the matching results, and is also equivalent to using the exponential windowing of the past data to estimate the current weight.

That is to say, for the distribution that is not selected or not matched, the weight remains unchanged; for the distribution that is selected or matched, the following formula is used:

μ _t+1 (x,y)=(1-α)u(x,y)+αI _t (x,y);

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\text{<span class="notranslate"> ;</span>}$

β=αN(X _t ,μ _k ,σ _k );

Make an update to the current distribution.

After updating the aforementioned mixed Gaussian model, by calculating w _k,t /σ _k,t of each Gaussian distribution in the aforementioned mixed Gaussian model, and sorting them, the ratio w _k,t /σ _k,t The larger the value, the larger w _k,t and the smaller σ _k,t , so the Gaussian distribution with higher ranking is more suitable for describing the background. Generally, the top M Gaussian distributions are selected as the background, and M is obtained by the following formula:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ></span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; )</span>$

Among them, the threshold T _R represents the weight of the distribution representing the background and the minimum proportion it occupies in the whole. M is the number of "best" distributions that achieve this ratio, i.e. the top m most probable distributions.

By matching each pixel point with the first m Gaussian distributions determined above, until a match is found, it is considered as a background point. If no Gaussian distribution matches _Xt , it is judged as a moving foreground point. The same judgment operation is performed on each pixel of the image frame, so as to obtain the moving foreground through the mixed Gaussian background model.

Step S104, acquiring the pixel coordinates of the moving foreground in the image frame.

The pixel coordinates are the positions of the pixels in the image frame. There are multiple image frames acquired by multiple cameras at the same time, and the moving foreground may have different pixel coordinates in each image frame.

Step S106, calculating the three-dimensional space coordinates of the moving foreground according to the pixel coordinates, focal lengths and space distances of multiple cameras.

In one embodiment, the number of cameras is two, and they are arranged horizontally. The acquired image frames are the left image frame (left) and the right image frame (right) respectively collected by the two cameras at the same time. In this embodiment, image frames may be collected by a binocular camera.

The steps of calculating the three-dimensional spatial coordinates of the moving foreground according to the pixel coordinates, the focal lengths and the spatial distances of multiple cameras can be specifically:

According to the formula:

Disparity＝X _left -X _right

Compute the inspections for the left image frame and the right image frame. Wherein, Disparity is a parallax, X _left is the horizontal coordinate of the moving foreground in the left image frame, and X _right is the horizontal coordinate of the moving foreground in the right image frame;

According to the formula:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; BX</span>}}_{\mathrm{<span\; class="notranslate">\; left</span>}}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; BY</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\end{array}\right.$

Calculate the three-dimensional space coordinates of the moving foreground; among them, (x _c , y _c , z _c ) are the three-dimensional space coordinates of the moving foreground, and x _c and y _c are the coordinates of the visual plane, z _c is the depth information, and B is two The horizontal distance between two cameras, f is the focal length of the camera, and Y is the vertical coordinate of the moving foreground in the left image frame and the right image frame (because the camera is set horizontally, the Y value of the pixel in the left image frame and the right image frame The Y value of the middle pixel is the same), and Disparity is the disparity between the left image frame and the right image frame.

Y）分别为同一物体分别在左右图像帧上产生的投影的像素坐标，（x_c，y_c，z_c）即计算得到该物体在三维空间中的实际坐标。For example, as shown in Figure 2, L and R are respectively the left image frame and the right image frame captured by the binocular camera set horizontally at the same time, and C _left and C _right are respectively the camera optical axes of the left and right cameras (through the focal length perpendicular to the camera mirror axis), B is the distance between the cameras (baseline distance), f _left and f _right are the focal lengths of the left and right cameras respectively, usually for the convenience of calculation, f _left and f _right can be set equal, both are f . (X _left , Y) and (

Y) are the pixel coordinates of the projection of the same object on the left and right image frames respectively, and (x _c , y _c , z _c ) are calculated to obtain the actual coordinates of the object in the three-dimensional space.

It should be noted that in other embodiments, the camera can also be placed vertically. When the cameras are placed vertically, B is the vertical distance between the cameras, and the three-dimensional space coordinates of the moving foreground can be obtained by exchanging X _left / X _right and Y.

In other embodiments, image frames may also be collected through more than two cameras placed in the space. Multiple groups of cameras can be selected from more than two cameras, and the number of cameras in each group is two. The three-dimensional space coordinates of the moving foreground can be calculated through the image frames simultaneously collected by each group of cameras, and then the calculated three-dimensional space coordinates of the moving foreground corresponding to each group of cameras are averaged, thereby improving the measurement accuracy.

Step S108, triggering an abnormal event of a moving foreground according to the three-dimensional space coordinates.

The abnormal event of the moving foreground is an event triggered when the moving foreground enters a preset dangerous area, and a corresponding alarm may be issued according to the triggered event.

In one embodiment, the step of triggering the abnormal event of the moving foreground according to the three-dimensional space coordinates may be specifically: determining whether the three-dimensional space coordinates are located in a preset dangerous area, and if so, triggering the abnormal event of the moving foreground.

For example, in the background of the image frame, the vicinity of dangerous facilities such as high-voltage wires can be pre-delineated as a dangerous area (also represented by three-dimensional space coordinates). When the high-voltage power line is close to the dangerous area, an abnormal event is triggered to warn pedestrians or notify relevant personnel to deal with it.

Further, the step of judging whether the three-dimensional space coordinates are located in the preset dangerous area may be specifically: acquiring the depth image corresponding to the image frame, the depth image is a grayscale image, and the coordinates of the pixels thereof correspond to the visual plane coordinates of the three-dimensional space coordinates, Its gray value corresponds to the depth information of the three-dimensional space coordinates; through the depth image, it is judged whether the three-dimensional space coordinates are located in the preset dangerous area.

For example, as shown in Fig. 3, Fig. 3 shows a depth image generated according to the three-dimensional space coordinates of the moving foreground in the image frame. In FIG. 3 , the pixel coordinates of the depth image are determined according to the visible plane coordinates in the three-dimensional space coordinates of the pixels, and then the gray value corresponding to the pixel coordinates is set as the depth information corresponding to the visible plane coordinates. For example, if the three-dimensional space coordinates of the moving foreground P are (x _p , y _p , z _p ), then the gray value at the pixel point (x _p , y _p ) of the depth image is γz _p , where γ is the preset The set coefficient can be determined according to the value range of z _p . For example, if the acquired z _p of all moving foregrounds ranges from 0 to 100, then γ can take a value of 255/100, that is, the minimum gray value can be 0, and the maximum gray value can be 255.

In this embodiment, before the step of triggering the abnormal event of the moving foreground, it is also possible to judge whether the moving foreground is a human gesture image according to the depth information, and if so, continue to execute the step of triggering the abnormal event of the moving foreground.

Further, the steps of judging whether the motion foreground is a human gesture image according to the depth information are:

According to the formula:

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; u</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Calculate the depth value of the moving foreground, where d(I,c) is the depth value of the pixel at coordinate c in the depth image I, u and v are the coordinates of any two points randomly selected within the threshold range, and z is the moving foreground Depth information on the above; obtain the training classifier of the preset human gesture image; judge whether the moving foreground is a human gesture image through a decision tree according to the depth value of the moving foreground and the training classifier.

In this embodiment, each pixel in the depth image I can be traversed, and then the depth value d of each pixel can be calculated by the above formula, and then the pixel can be judged according to the depth value d of the traversed pixel through the decision tree Whether it belongs to the edge area of the moving foreground (that is, to judge whether the depth value d is in the corresponding threshold interval), so that the relative positions of all pixel coordinates in the moving foreground in the moving foreground can be obtained. For example, if pixel a corresponds to the edge then The relative position attribute value of the pixel point can be set to 1, and if the pixel point b corresponds to the inner depth of the moving foreground (that is, not the edge), the relative position attribute value of the pixel point can be set to 0. Finally, all pixel coordinates can be input into the training classifier to judge whether the motion foreground is a human gesture image, and similarly, the range of pixel coordinates belonging to the human gesture image in the motion foreground can also be obtained.

The training classifier can be generated through training samples in advance. For example, a training classifier may be generated according to multiple depth images having human gesture images, so as to obtain a classification kernel function of the training classifier.

In one embodiment, as shown in FIG. 4 , a video surveillance system includes a multi-eye camera 102, a moving foreground extraction device 104, a coordinate conversion device 106, and an abnormal event triggering device 108, wherein:

The multi-eye camera 102 is used to collect image frames.

A moving foreground extracting device 104 is configured to extract a moving foreground according to image frames.

A multi-camera is an image acquisition device with multiple cameras. The multi-camera can collect multi-angle images of the same object through its own multiple cameras at the same time.

The data form of the images collected by the multi-camera is continuous image frames. A moving foreground is a region of pixels that changes in successive image frames.

In one embodiment, the moving foreground extracting device 104 can be used to extract the moving foreground of the image frame by background difference according to the mixed Gaussian model.

In this embodiment, the mixed Gaussian model is pre-defined as:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

$\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\frac{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span>}^{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ;</span>$

${\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; ;</span>$

Among them, X _t is the pixel value of the pixel point (x, y) at time t (since the sampling frequency of the sampling image frame is a fixed value when the camera is shooting video images, t can also be the number of frames), and K is the preset The number of mixed Gaussian models (usually 3 to 5), w _{k, t} is the weight of the kth Gaussian distribution at the preset time t, μ _{k, t} is the mean value of the kth Gaussian distribution at time t , ∑ _{k, t} is the covariance matrix of the kth Gaussian distribution at time t, N is the probability density function of Gaussian distribution, is the variance of the kth Gaussian distribution, and E is the expected value of the Gaussian distribution.

After the image frame is acquired, the moving foreground extraction device 104 can be used to update the Gaussian model according to the current moment. The online K-means algorithm can be used to approximate the parameters. The online K-means algorithm matches the pixel at the current moment with K Gaussian distributions. If the matching is successful, the mean and variance of the distribution are updated, and the weight of the distribution is increased; if there is no match, a new distribution is generated to replace it. The term with the smaller weight in the existing mixture distribution.

The moving foreground extraction device 104 can be used to arrange the aforementioned K Gaussian distributions according to the ratio of weight and variance from large to small, and then select the Gaussian distribution closest to the distribution mean as the matching Gaussian distribution, namely:

${\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&lambda;\&sigma;</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; otherwise</span>}\end{array}\right.$

Among them, M _{k, t} is the matching coefficient, that is, the k value when M _{k, t} = 1 is the k value of the finally selected matching Gaussian model, X _t is the pixel value of the pixel point (x, y) at time t, μ _k,t is the mean value of the kth Gaussian distribution at time t, σ is the variance, and λ is the preset coefficient.

If no matching distribution is found for the current pixel among the K distributions, the least likely distribution is replaced by a new distribution. The mean of the new distribution is set to the current pixel value, with larger variance and smaller weight.

The moving foreground extraction device 104 can also be used to adjust the weights of the K distributions at time t according to the selected Gaussian model, which can be passed through the formula:

w _k,t+1 ＝(1-α)w _k,t +αM _k,t

w _k,t+1 =w _k,t +α(M _k,t -w _k,t )

Adjust the weights. Among them, α is not a preset coefficient, M _{k, t} is the aforementioned matching coefficient, and w _{k, t} is the weight of the kth Gaussian distribution at time t. After adjusting the weights, the sum of the weights remains the same and is still 1. The update process is equivalent to performing a causal second-order low-pass filter on the matching results, and is also equivalent to using the exponential windowing of the past data to estimate the current weight.

That is to say, for the distribution that is not selected or not matched, the weight remains unchanged; for the distribution that is selected or matched, the following formula is used:

μ _t+1 (x,y)=(1-α)u(x,y)+αI _t (x,y);

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\text{<span class="notranslate"> ;</span>}$

β=αN(X _t ,μ _k ,σ _k );

Make an update to the current distribution.

The moving foreground extraction device 104 can also be used to update the aforementioned mixed Gaussian model, by calculating w _k,t /σ _k,t of each Gaussian distribution in the aforementioned mixed Gaussian model, and sorting them, the ratio w _{k ,t} /σ _k,t is larger, it means that w _k,t is larger and σ _{k,t is} smaller, so the Gaussian distribution with higher ranking is more suitable for describing the background. Generally, the top M Gaussian distributions are selected as the background, and M is obtained by the following formula:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ></span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; )</span>$

Among them, the threshold T _R represents the weight of the distribution representing the background and the minimum proportion it occupies in the whole. M is the number of "best" distributions that achieve this ratio, i.e. the top m most probable distributions.

The moving foreground extraction device 104 can also be used to match each pixel point with the above-mentioned first m Gaussian distributions one by one until a match is found, then it is considered as a background point, if no Gaussian distribution matches _Xt , it is determined as Movement prospect. The same judgment operation is performed on each pixel of the image frame, so as to obtain the moving foreground through the mixed Gaussian background model.

The coordinate conversion device 106 is configured to obtain the pixel coordinates of the moving foreground in the image frame, and calculate the three-dimensional space coordinates of the moving foreground according to the pixel coordinates, the focal length and the spatial distance of the multi-camera.

The pixel coordinates are the positions of the pixels in the image frame. There are multiple image frames acquired by the multi-eye camera 102 at the same time, and the moving foreground may have different pixel coordinates in each image frame.

In one embodiment, the multi-camera 102 is a binocular camera arranged horizontally. The image frames acquired by the multi-camera 102 are the left image frame (left) and the right image frame (right) respectively acquired by the binocular cameras at the same time.

As shown in Figure 5, where left and right are the left and right lenses of the binocular camera set horizontally, respectively, and B is the distance between the focal points of the left and right lenses, also called the baseline distance. It should be noted that left and right are only used to distinguish the left and right lenses, and do not limit the specific positions of the lenses.

In this embodiment, the coordinate conversion device 106 can also be used according to the formula:

Disparity＝X _left -X _right

Compute the observations for the left image frame and the right image frame. Wherein, Disparity is the parallax, X _left is the horizontal coordinate of the moving foreground in the left image frame, and X _right is the horizontal coordinate of the moving foreground in the right image frame.

Coordinate conversion means 106 can also be used according to the formula:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; BX</span>}}_{\mathrm{<span\; class="notranslate">\; left</span>}}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; BY</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; Disparity</span>}}\end{array}\right.$

Calculate the three-dimensional space coordinates of the moving foreground; among them, (x _c , y _c , z _c ) are the three-dimensional space coordinates of the moving foreground, and x _c and y _c are the coordinates of the visual plane, z _c is the depth information, and B is two The horizontal distance between two cameras, f is the focal length of the camera, and Y is the vertical coordinate of the moving foreground in the left image frame and the right image frame (because the camera is set horizontally, the Y value of the pixel in the left image frame and the right image frame The Y value of the middle pixel is the same), and Disparity is the disparity between the left image frame and the right image frame.

Y）和（

Y）分别为同一物体分别在左右图像帧上产生的投影的像素坐标，（x_c，y_c，z_c）即计算得到该物体在三维空间中的实际坐标。For example, as shown in Figure 2, L and R are respectively the left image frame and the right image frame captured by the binocular camera set horizontally at the same time, and C _left and C _right are respectively the camera optical axes of the left and right cameras (through the focal length perpendicular to the camera mirror axis), B is the distance between the cameras (baseline distance), f _left and f _right are the focal lengths of the left and right cameras respectively, usually for the convenience of calculation, f _left and f _right can be set equal, both are f . (

Y) and (

Y) are the pixel coordinates of the projection of the same object on the left and right image frames respectively, and (x _c , y _c , z _c ) are calculated to obtain the actual coordinates of the object in the three-dimensional space.

It should be noted that in other embodiments, the binocular cameras can also be arranged vertically. When the cameras are placed vertically, B is the vertical distance between the cameras, and the three-dimensional space coordinates of the moving foreground can be obtained by exchanging X _left / X _right and Y.

In other embodiments, image frames may also be collected through multiple lenses of the multi-eye camera 102 . The multiple lenses of the multi-eye camera 102 can be divided into multiple groups, and the number of each group is two. The three-dimensional space coordinates of the moving foreground can be calculated through the image frames collected simultaneously by each group of shots, and the calculated three-dimensional space coordinates of the moving foreground corresponding to each group of shots are averaged, thereby improving the measurement accuracy.

The abnormal event triggering device 108 is configured to trigger an abnormal event of the moving foreground according to the three-dimensional space coordinates.

The abnormal event of the moving foreground is an event triggered when the moving foreground enters a preset dangerous area, and a corresponding alarm may be issued according to the triggered event.

In one embodiment, the abnormal event triggering device 108 can be used to determine whether the three-dimensional space coordinates are located in a preset dangerous area, and if so, trigger an abnormal event of the moving foreground.

For example, in the background of the image frame, the vicinity of dangerous facilities such as high-voltage wires can be pre-delineated as a dangerous area (also represented by three-dimensional space coordinates). When the high-voltage power line is close to the dangerous area, an abnormal event is triggered to warn pedestrians or notify relevant personnel to deal with it.

Further, the abnormal event triggering device 108 can also be used to obtain the depth image corresponding to the image frame, the depth image is a grayscale image, the coordinates of its pixels correspond to the visual plane coordinates of the three-dimensional space coordinates, and its grayscale value corresponds to the coordinates of the three-dimensional space coordinates. Depth information; determine whether the three-dimensional space coordinates are located in the preset dangerous area through the depth image.

For example, as shown in Fig. 3, Fig. 3 shows a depth image generated according to the three-dimensional space coordinates of the moving foreground in the image frame. In FIG. 3 , the pixel coordinates of the depth image are determined according to the visible plane coordinates in the three-dimensional space coordinates of the pixels, and then the gray value corresponding to the pixel coordinates is set as the depth information corresponding to the visible plane coordinates. For example, if the three-dimensional space coordinates of the moving foreground P are (xp _, y _p , z _p ), then the gray value at the pixel point (x _p , y _p ) of the depth image is γz _p , where γ is the preset The coefficient of can be determined according to the value range of z _p . For example, if the acquired z _p of all moving foregrounds ranges from 0 to 100, then γ can take a value of 255/100, that is, the minimum gray value can be 0, and the maximum gray value can be 255.

In this embodiment, the abnormal event triggering device 108 can also be used to determine whether the moving foreground is a human gesture image according to the depth information, and if so, execute an abnormal event that triggers the moving foreground.

Further, the abnormal event triggering device 108 can also be used according to the formula:

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; u</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Calculate the depth value of the moving foreground, where d(I,c) is the depth value of the pixel at coordinate c in the depth image I, u and v are the coordinates of any two points randomly selected within the threshold range, and z is the moving foreground Depth information on the above; obtain the training classifier of the preset human gesture image; judge whether the moving foreground is a human gesture image through a decision tree according to the depth value of the moving foreground and the training classifier.

In this embodiment, the abnormal event triggering device 108 can be used to traverse each pixel in the depth image I, and then calculate the depth value d of each pixel through the above formula, and then use the decision tree to The depth value d judges whether the pixel belongs to the edge area of the moving foreground (that is, judges whether the depth value d is in the corresponding threshold range), so that the relative positions of all pixel coordinates in the moving foreground can be obtained, for example, If the pixel point a corresponds to the edge, the relative position attribute value of the pixel point can be set to 1; if the pixel point b corresponds to the inner depth of the moving foreground (that is, not the edge), the relative position attribute value of the pixel point can be set to is 0. Finally, all pixel coordinates can be input into the training classifier to judge whether the motion foreground is a human gesture image, and similarly, the range of pixel coordinates belonging to the human gesture image in the motion foreground can also be obtained.

The training classifier can be generated through training samples in advance. For example, a training classifier may be generated according to multiple depth images having human gesture images, so as to obtain a classification kernel function of the training classifier.

The above video surveillance method and device collects image frames through multi-eye cameras, extracts the moving foreground according to the image frames, then converts the pixel coordinates of the moving foreground in the image frames into the three-dimensional space coordinates of its actual position, and according to the motion The three-dimensional space coordinates of the foreground trigger abnormal events. Compared with the traditional technology, not only the abnormal events can be triggered according to the position of the moving foreground on the two-dimensional plane, but also the depth information can be obtained from the converted three-dimensional space coordinates of the moving foreground ( The distance from the camera) triggers abnormal events, which makes the triggering of abnormal events more accurate, thereby improving security.

The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (14)

1. a video frequency monitoring method, comprising:

Gather the picture frame of a plurality of cameras, according to described picture frame, extract sport foreground;

Obtain the pixel coordinate of described sport foreground in described picture frame;

According to the focal length of described pixel coordinate, a plurality of cameras and space length, calculate the three dimensional space coordinate of described sport foreground;

According to described three dimensional space coordinate, trigger the anomalous event of described sport foreground.

2. video frequency monitoring method according to claim 1, is characterized in that, the described step according to described picture frame extraction sport foreground is:

According to mixed Gauss model, by background difference, extract the sport foreground of described picture frame.

3. video frequency monitoring method according to claim 1, is characterized in that, the quantity of described camera is 2, and is horizontally disposed with;

The described picture frame getting is left picture frame and the right picture frame being gathered respectively by described 2 cameras the same time;

The described step of calculating the three dimensional space coordinate of described sport foreground according to the focal length of described pixel coordinate, a plurality of cameras and space length is:

According to formula:

Disparity＝X _left-X _right

Calculate the inspection of described left picture frame and right picture frame, wherein, Disparity is parallax, X _leftfor the horizontal coordinate of sport foreground in described left picture frame, X _rightfor the horizontal coordinate of sport foreground in described right picture frame;

According to formula:

$\left\{\begin{array}{c}{x}_c=\frac{{\mathrm{BX}}_{\mathrm{left}}}{\mathrm{Disparity}}\\ y_c=\frac{\mathrm{BY}}{\mathrm{Disparity}}\\ z_c=\frac{\mathrm{Bf}}{\mathrm{Disparity}}\end{array}\right.$

Calculate the three dimensional space coordinate of described sport foreground; Wherein, (x _c, y _c, z _c) be the three dimensional space coordinate of described sport foreground and x _cand y _cfor visible planar coordinate, z _cfor depth information, B is the horizontal range between described two cameras, the focal length that f is described camera, and Y is that sport foreground obtains vertical coordinate in described left picture frame and described right picture frame, Disparity is the parallax of left picture frame and right picture frame.

4. video frequency monitoring method according to claim 3, is characterized in that, the described step that triggers the anomalous event of described sport foreground according to described three dimensional space coordinate is:

Judge that whether described three dimensional space coordinate is positioned at default deathtrap, if so, triggers the anomalous event of described sport foreground.

5. video frequency monitoring method according to claim 4, is characterized in that, the described step that judges whether described three dimensional space coordinate is positioned at default deathtrap is:

Obtain the depth image that described picture frame is corresponding, described depth image is gray-scale map, the visible planar coordinate of the coordinate corresponding three-dimensional space coordinates of its pixel, the depth information of its gray value corresponding three-dimensional space coordinates;

By described depth image, judge whether described three dimensional space coordinate is positioned at default deathtrap.

6. video frequency monitoring method according to claim 3, is characterized in that, before the step of the anomalous event of the described sport foreground of described triggering, also comprises:

According to described depth information, judge whether described sport foreground is human body images of gestures, if so, continue the step of the anomalous event of the described sport foreground of the described triggering of execution.

7. video frequency monitoring method according to claim 6, is characterized in that, describedly according to described depth information, judges that whether described sport foreground is that the step of human body images of gestures is:

According to formula:

$d(I,c)=z(c+\frac{u}{z\left(c\right)})-z(c+\frac{v}{z\left(c\right)})$

Calculate the depth value of described sport foreground, wherein d (I, c) is the depth value of coordinate c place pixel in depth image I, the coordinate that u and v are any two points chosen at random in threshold range, and z is the depth information in sport foreground;

Obtain the training classifier of default human body images of gestures;

According to the depth value of described sport foreground and described training classifier, by decision tree, judge whether described sport foreground is human body images of gestures.

8. a video monitoring system, is characterized in that, comprising:

Many orders camera, for acquired image frames;

Sport foreground extraction element, for extracting sport foreground according to described picture frame;

Coordinate transformation device, for obtaining described sport foreground at the pixel coordinate of described picture frame, the three dimensional space coordinate that calculates described sport foreground according to the focal length of described pixel coordinate, many orders camera and space length;

Anomalous event trigger equipment, for triggering the anomalous event of described sport foreground according to described three dimensional space coordinate.

9. video monitoring system according to claim 8, is characterized in that, described sport foreground extraction element is also for extracting the sport foreground of described picture frame by background difference according to mixed Gauss model.

10. video monitoring system according to claim 8, is characterized in that, described many orders camera is horizontally disposed binocular camera shooting head;

The described picture frame getting is left picture frame and the right picture frame being gathered by described binocular camera shooting head the same time;

Described coordinate transformation device is also for according to formula:

Disparity＝X _left-X _right

Calculate the inspection of described left picture frame and right picture frame, wherein, Disparity is parallax, X _leftfor the horizontal coordinate of sport foreground in described left picture frame, X _rightfor the horizontal coordinate of sport foreground in described right picture frame, and according to formula:

$\left\{\begin{array}{c}{x}_c=\frac{{\mathrm{BX}}_{\mathrm{left}}}{\mathrm{Disparity}}\\ y_c=\frac{\mathrm{BY}}{\mathrm{Disparity}}\\ z_c=\frac{\mathrm{Bf}}{\mathrm{Disparity}}\end{array}\right.$

Calculate the three dimensional space coordinate of described sport foreground; Wherein, (x _c, y _c, z _c) be the three dimensional space coordinate of described sport foreground and x _cand y _cfor visible planar coordinate, z _cfor depth information, B is the horizontal range between described two cameras, the focal length that f is described camera, and Y is that sport foreground obtains vertical coordinate in described left picture frame and described right picture frame, Disparity is the parallax of left picture frame and right picture frame.

11. video monitoring systems according to claim 10, is characterized in that, described anomalous event trigger equipment is also for judging that whether described three dimensional space coordinate is positioned at default deathtrap, if so, triggers the anomalous event of described sport foreground.

12. video monitoring systems according to claim 11, is characterized in that, described anomalous event trigger equipment also for

Obtain the depth image that described picture frame is corresponding, described depth image is gray-scale map, the visible planar coordinate of the coordinate corresponding three-dimensional space coordinates of its pixel, the depth information of its gray value corresponding three-dimensional space coordinates, and judge by described depth image whether described three dimensional space coordinate is positioned at default deathtrap.

13. video monitoring systems according to claim 10, is characterized in that, before the step of the anomalous event of the described sport foreground of described triggering, also comprise:

According to described depth information, judge whether described sport foreground is human body images of gestures, if so, continue the step of the anomalous event of the described sport foreground of the described triggering of execution.

14. video monitoring systems according to claim 13, is characterized in that, described anomalous event trigger equipment is also for according to formula:

$d(I,c)=z(c+\frac{u}{z\left(c\right)})-z(c+\frac{v}{z\left(c\right)})$

Calculate the depth value of described sport foreground, wherein d (I, c) is the depth value of coordinate c place pixel in depth image I, the coordinate that u and v are any two points chosen at random in threshold range, and z is the depth information in sport foreground; Obtain the training classifier of default human body images of gestures; According to the depth value of described sport foreground and described training classifier, by decision tree, judge whether described sport foreground is human body images of gestures.

Images