KR20090030142A - Image-based Human Blob Tracking Method - Google Patents

Abstract

The present invention relates to a system and method for continuously tracking a blob of people appearing on a screen in a continuous image in an image based video surveillance system captured by a video camera.

An image-based human blob tracking system according to the present invention includes a frame extractor for extracting a frame from a video, a blob information storage unit for storing information about the frames of each frame, and a person in the current frame extracted by the frame extractor. A blob extraction unit for extracting an area into blobs, a distance calculator for calculating a distance between the blobs extracted from the current frame and the blobs extracted from the previous frame stored in the blob information storage unit, and the distance calculator The blob state detection unit classifies the blobs extracted from the current frame, allocates the set of people, and stores the blob information stored in the blob information storage unit by using the distance between the blobs and the blob information of the previous frame stored in the blob information storage unit. Include.

Description

Image-based human blob tracking system and method

The present invention relates to a system and method for continuously tracking a blob of people appearing on a screen in a continuous image in an image based video surveillance system captured by a video camera.

As image-based tracking technology, techniques such as object tracking, human tracking, and hand tracking have been proposed. In the prior art, the movement path is predicted based on the position of an object (collectively, hand and human) appearing on the screen, and the object is tracked using the position information of the object. Thus, when several objects overlap and fall on the screen, a problem arises in that the tracking object is reversed without being able to track the existing object as it is.

In addition, in the hand tracking and human tracking technology, only one person is mainly tracked, and a lot of researches on extracting one person or one hand and analyzing the shape have been conducted. Little research has been done to analyze the shape of the blobs.

The present invention is to solve the problems of the prior art, a system and method for simultaneously extracting the shape information of the human blob while simultaneously tracking several people in the situation where many people appear, disappear and overlap based on the video camera image Suggest.

In order to achieve the above object, an image-based human blob tracking system according to the present invention includes a frame extractor for extracting a frame from a video, a blob information storage unit for storing information about the frames of each frame, and the frame extractor. A blob extraction unit for extracting a human region from the current frame extracted from the unit as blobs, and a distance calculator for calculating the distance between the blobs extracted from the current frame and the blobs extracted from the previous frame stored in the blob information storage unit. And classifying the blobs extracted from the current frame by using the distance between the blobs calculated by the distance calculator and the blob information of the previous frame stored in the blob information storage unit, assigning the set of people, and storing the blob information. Characterized in that it comprises a blob state to be stored in the wealth.

In addition, an image-based human blob tracking method according to the present invention includes a frame extraction step of extracting a frame from a video, a blob extraction step of extracting a human region from the current frame extracted in the frame extraction step as blobs, and A distance calculation step of calculating the distance between the blobs extracted in the current frame and the blobs extracted in the previous frame, and extracted from the current frame by using the distance between the blobs calculated in the distance calculation step and the blob information in the previous frame And a blob status grasping step of classifying the blobs' status and allocating person set information, and a blob information storing step of storing blob information extracted from the current frame.

According to the present invention, there is an advantage in that it can accurately and accurately track the actions and movements of several people included in the video and use them in action recognition.

Hereinafter, an image-based human blob tracking system and method according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a functional block diagram of an image-based human blob tracking system according to the present invention.

The image-based human blob tracking system includes an image acquisition unit 11 for acquiring a video, a frame extractor 12 for extracting a frame from the video, and a blob information storage unit for storing information about the blobs for each frame. 13, a blob extractor 14 for extracting human blobs from the current frame extracted from the frame extractor 12, and a blob extracted from the current frame and the previous frame stored in the blob information storage unit 13; The current frame using the distance calculator 15 for calculating the distance between the extracted blobs, the distance between the blobs calculated by the distance calculator 15 and the blob information of the previous frame stored in the blob information storage unit 13. The blob state detection unit 16 for classifying the state of the blobs extracted from and stored in the blob information storage unit (13).

The distance calculator 15 calculates the weight center, size, and hue histogram distribution of each blob, and calculates the distance between the blobs using the blobs.

The blob state detection unit 16 divides the blob into five states, namely, appearance, disappearing, tracking continuation, merging, and splitting. The appearing state means that the blob is newly entering a specific region of the image, and the disappearing state means that the blob leaves the specific region of the image, and the tracking continuation state It is a state moving within a specific area, merging is a state in which two or more blobs are merged into one blob in the current frame, and a splitting state is one in the previous frame. The blob is split into two or more blobs in the current frame.

The blob state detecting unit 16 calculates the distance between the blobs extracted in the current frame and the blobs extracted in the previous frame to determine whether there are blobs matching each other. As a result of the determination, if there is a matching blob, it is determined whether the blob is out of a specific area. If the blob is in an exited state out of a specific area, it is determined as an exited state and assigned and stored with matching person information. If the blob is in a specific area, it is determined as a tracking state and assigned and stored as matching person information.

On the other hand, if the blob of the current frame and the blob of the previous frame do not match, it is sequentially checked whether it is merging, splitting, or appearing. In the merged state, the person information allocated to the blobs before the merge of the previous frame is added and stored as the new merged blob information. In the separated state, appropriate blob information is allocated to the separated blob and stored using the blob information of the previous frame. In the case of appearance, new person information is allocated to the appeared blob and stored.

In order to recognize individual actions when several people are moving at the same time, it is necessary to distinguish blobs that are considered to be humans in sequential images captured by a video camera, and to determine relationships between blobs in sequential frames. To simplify the problem of tracking multiple blobs in a system that wants to recognize the actions of different people, the following assumptions are made:

1. Two or more people do not appear or disappear on the screen at the same time. When people appear on the screen, they appear and disappear one by one.

2. People do not disappear suddenly in the center of the screen. Considering the movement speed of the person and the viewpoint of the camera, it is assumed that the person in the center of the screen does not disappear out of the screen.

3. Since people must be identified without additional information such as tags, assume that people have different appearances (colors), and use the appearance model to distinguish people.

Referring to FIG. 2, an image-based human blob tracking method according to an embodiment of the present invention will be described.

First, the video acquisition unit 11 acquires a video (step S21).

The frame extractor 12 extracts an image frame with respect to the obtained video (step S22).

The blob extractor 14 extracts a human area from the image frame extracted by the frame extractor 12 as a blob (step S23).

When a set of blobs from the image frame as t B _t, B _t a is expressed as Equation (1).

B_t = { | i는 정수, 1≤i≤n-1}

B _t = {| i is an integer, 1≤i≤n-1}

는 프레임 t에서의 i번째 블랍을 나타내며, 배경픽셀(background pixels)를 제외한 전경픽셀(foreground pixel)들의 하나의 연결성분(connected component)으로 구해진다. 보통 하나의 블랍

은 한 사람을 의미하지만(조명, 그림자 등의 잡음에 의해 분할된 블랍들은 모두 제거되었다고 가정한다), 두 사람 이상이 화면상에서 중첩되는 경우에는 하나의 블랍

이 두 사람 이상을 의미하기도 한다. 이와 같이 한 블랍이 의미하는 사람들의 집합을

로 표시한다.Where n is the total number of blobs,

Denotes the i-th blob in frame t, and is obtained as one connected component of foreground pixels except for background pixels. Usually one blob

Means one person (assuming all blobs split by noise such as lighting, shadows, etc. are removed), but one blob if two or more people overlap on screen

It also means more than two people. This blob means a set of people

To be displayed.

(여기서, 0≤i≤B_t의 원소의 개수, 0≤j≤ B_{t -1}의 원소의 개수이다.)를 계산한다(단계 S24). 두 블랍간 거리

는 아래의 수학식 2와 같이 구한다.The distance calculator 15 must determine the relationship of each blob in the sequential frames so that each blob can be distinguished from all blobs included in the blob set B _t of the frame t, and the previous frame t−. The distance between all the blobs in the blob set B _{t -1} of ₁

(Here, the number of elements of 0 ≦ _i ≦ B _t and the number of elements of 0 ≦ j ≦ B _{t −1} ) is calculated (step S24). Two Blobgan Streets

Is obtained as in Equation 2 below.

는 각 요소의 가중치를 나타내며, 0≤

≤1, 0≤

≤1, 0≤

≤1,

의 조건을 만족한다.Where D _size is the normalized size difference between the two blobs, D _cent is the normalized distance between the centers of gravity of the two blobs, and D _hue represents the normalized distance of the hue histogram of the two blobs. Also,

Represents the weight of each element, 0≤

≤1, 0≤

≤1, 0≤

≤1,

Satisfies the conditions.

The blob state detection unit 16 determines the state of each blob of the blob set of the frame t and assigns a person (step S25). The blob state detection routine will be described in detail with reference to the flowchart of FIG.

에 대해

가 최소가 되는 j를 구하여 j_min이라 하고(단계 S31),

이 임계값보다 작으면(단계 S32), 두 블랍(

,

)을 매칭하고(단계 S33), 블랍

에 할당된 사람집합

을 블랍

에 할당한다(단계 S34).Any one blob at frame t

About

Let j be the minimum and call it j _min (step S31),

If less than this threshold (step S32), two blobs (

,

) (Step S33), and the blob

Set of people assigned to

Blob

(Step S34).

이 영상의 특정영역 내에 존재하면(단계 S35), 이 블랍

를 추적(tracking continuation) 상태로 판단하고(단계 S36), 이 블랍

이 영상의 특정영역 내에 존재하지 않으면(단계 S35) 퇴장(disappearing) 상태로 판단한다(단계 S37).This blob

If present in a specific area of this image (step S35), this blob

Is determined to be a tracking continuation state (step S36), and the blob

If it is not present in the specific area of the image (step S35), it is determined to be in a disappearing state (step S37).

이 임계값보다 작지 않으면, 프레임 t에서의 임의의 한 블랍

와 매칭되는 프레임 t-1에서의 블랍이 없는 것으로 인지한다. 이 경우, 블랍

은 병합(merging) 상태, 분리(splitting) 상태, 등장(appearing) 상태 중 한 상태가 된다. 아래에서는 블랍

을 병합 상태, 분리 상태, 등장 상태로 판단하는 과정을 설명한다.On the other hand, in step S32

If it is not less than this threshold, any one blob in frame t

Recognize that there is no blob in frame t-1 that matches. In this case, the blob

Is one of a merging state, a splitting state, and an appearing state. Blob from below

It describes the process of determining the merge state, separation state, appearance state.

와 B_{t -1}의 관계를 도시한 도면이다.Figure 4 determines the merging and separation of the blobs

And B _{t -1} .

와 프레임 t-1의 블랍 집합 B_{t -1}의 겹치는 블랍들의 집합을 구하여 그 중 한 원소를 b^x라고 가정한다(단계 S38). 이를 수식으로 표현하면 수학식 3과 같다.Blob of frame t

And a set of overlapping blobs of the blob set B _{t -1} of frame t-1 and one of them is assumed to be b ^x (step S38). If this is expressed as an equation, Equation 3 is obtained.

b^x ∈ {B_{t -1} ∩ }

b ^x ∈ {B _{t -1} ∩}

와 유사하면

를 병합상태로 판단하고 b^x가

와 유사하면

를 분리상태로 판단한다(단계 S40).

가 병합상태로 판단된 경우에는,

가 의미하는 사람집합은 프레임 t-1의 병합전 각 블랍들에 할당된 사람들의 집합의 합집합으로 구한다.

가 분리상태로 판단된 경우에는,

가 의미하는 사람집합은 사람들의 외관모델들을 이용하여 모든 확률을 계산하여 가장 높은 확률을 가지는 사람을 할당한다.At this time, And B _{t −1} are related (step S39), and b ^x is as shown in FIG.

Is similar to

Is determined to be a merged state and b ^x is

Is similar to

It is determined that the separation state (step S40).

If is determined to be merged,

Means the union of the set of people assigned to each blob before the merge of frame t-1.

If is determined to be in the disconnected state,

Means that the person with the highest probability is assigned by calculating all the probabilities using people's appearance models.

가 B_{t -1}과 관련이 없으면, 이 블랍

를 등장 상태로 판단한다(S41). 이 경우에는 새로운 식별자를 가지는 새로운 사람집합을 할당한다.Blob in Step S39

Is not related to B _{t -1} , this blob

It is determined that the appearance state (S41). In this case, we assign a new set of people with a new identifier.

Although the technical spirit of the present invention has been described above with the accompanying drawings, it is intended to exemplarily describe the best embodiment of the present invention, but not to limit the present invention. In addition, it is obvious that any person skilled in the art may make various modifications and imitations without departing from the scope of the technical idea of the present invention.

This invention is a technology corresponding to the visual processing software of the Visual Surveillance System, and can be utilized in the technical fields such as ubiquitous computing, human-robot interaction, and next-generation interface.

1 is a functional block diagram of an image-based human blob tracking system according to the present invention;

2 is a schematic operation flowchart of an image-based human blob tracking method according to the present invention;

3 is a detailed operation flowchart of the blob state grabbing routine according to the present invention;

와 B_{t -1}의 관계를 도시한 도면이다.Figure 4 determines the merging and separation of the blobs

And B _{t -1} .

     <Brief description of symbols for the main parts of the drawings>

11: image acquisition unit 12: frame extraction unit

13: blob information storage unit 14: blob extraction unit

15: distance calculation unit 16: blob state detection unit

Claims (12)

A frame extractor which extracts a frame from a video; A blob information storage unit for storing information about the blobs of each frame; A blob extracting unit extracting a human region into blobs in the current frame extracted by the frame extracting unit; A distance calculator for calculating a distance between the blobs extracted from the current frame and the blobs extracted from the previous frame stored in the blob information storage unit; Classify the blobs extracted from the current frame by using the distance between the blobs calculated by the distance calculator and the blob information of the previous frame stored in the blob information storage unit, and assigns the set of people to the blob information storage unit. An image-based human blob tracking system comprising a blob state detection unit for storing. The image-based human blob tracking system of claim 1, wherein the distance calculator calculates a distance between the blobs using a center of gravity, size, and hue histogram distribution of the blobs. The method of claim 1, wherein the blob state detection unit classifies the blobs into classifications such as appearing, disappearing, tracking continuation, merging, and splitting. Image-based human blob tracking system. The method of claim 3, wherein the blob state detection unit, For any blob of the current frame, if the blob of the current frame and the blobs of the previous frame are less than or equal to a threshold, the blob of the current frame and the blob of the minimum distance are matched, and the blob of the current frame The image-based human blob tracking system of claim 1, wherein the blob state of the current frame is determined as a tracking state or an exit state according to the position of the frame. The image-based human blob tracking system of claim 4, wherein if the blob of the current frame matches the blob of the minimum distance, the set of persons assigned to the blob of the minimum distance is allocated to the blob of the current frame. . The method of claim 3, wherein the blob state detection unit, For any blob of the current frame, if the minimum distance between the blob of the current frame and the blobs of the previous frame is greater than a threshold, the blobs of the current frame and the blobs overlapping the blob set of the previous frame are determined to determine the current frame. If a blob of is related to the blob set of the previous frame, it is determined to be a merged state or a separated state, and if the blob of the current frame is not related to the blob set of the previous frame, the image-based human blob tracking system. A frame extraction step of extracting a frame from a video; A blob extraction step of extracting a human region from the current frame extracted in the frame extraction step into blobs; Calculating a distance between the blobs extracted in the current frame and the blobs extracted in the previous frame; A blob state grasp step of classifying the blobs extracted from the current frame and assigning a set of people using the distance between the blobs calculated in the distance calculation step and the blob information in the previous frame; An image-based human blob tracking method comprising a blob information storing step of storing blob information extracted from the current frame. The image-based human blob tracking method according to claim 7, wherein the distance calculating step calculates the distance between the blobs using the center of gravity, the size, and the hue histogram distribution of the blobs. The method of claim 7, wherein the identifying the blobs comprises classifying the blobs into an appearing, disappearing, tracking continuation, merging, and splitting state. Image-based human blob tracking method. The method of claim 9, wherein the blob state detection step, For any blob of the current frame, if the minimum distance between the blob of the current frame and the blobs of the previous frame is less than or equal to a threshold, the blob of the current frame and the blob of the minimum distance are matched, and the blob of the minimum distance An image-based human blob tracking method comprising assigning a set of persons assigned to a blob of the current frame and determining a blob state of the current frame as a tracking state or an exit state according to the position of the blob of the current frame. . The image-based human blob tracking method according to claim 10, wherein if the blob of the current frame matches the blob of the minimum distance, the set of people assigned to the blob of the minimum distance is allocated to the blob of the current frame. . The method of claim 10, wherein the blob state detection step, For any blob of the current frame, if the minimum distance between the blob of the current frame and the blobs of the previous frame is greater than a threshold, the blobs of the current frame and the blobs overlapping the blob set of the previous frame are determined to determine the current frame. If a blob of is related to the blob set of the previous frame, it is determined to be a merged state or a separated state, and if the blob of the current frame is not related to the blob set of the previous frame, the image-based human blob tracking Way.

Images