WO2017150899A1 - Object reidentification method for global multi-object tracking - Google Patents

Abstract

The present invention relates to an object reidentification method for global multi-object tracking, which is directed to global multi-object tracking for cameras having no view sharing therebetween, the method comprising the steps of: extracting an image of one or more objects included in an image received from the outside; estimating angular coordinates for each of first patches which are groups of pixels constituting the extracted image of the objects; comparing the estimated angular coordinates with angular coordinates of second patches which are group of pixels constituting a pre-stored object image; and reidentifying the one or more objects on the basis of the result of the comparison.

Description

Object Reidentification Method for Global Multiple Object Tracking

The present invention relates to an object re-identification method for global multi-object tracking, and more particularly, to an object re-identification method for global multi-object tracking for global multi-object tracking in cameras without view sharing. .

For security purposes, such as detecting and prosecuting crimes, the number of cameras used for video surveillance is on the rise. As a result, the demand for algorithms for intelligent video analysis is increasing because it can increase the efficiency of video surveillance and support crime prevention. Re-identification is the task of recognizing pedestrians observed through a network of cameras that do not share their view. This is important because the algorithms for intelligent video analysis applied to one camera can be made to work on multiple cameras with no field of view.

Interest in person re-identification has increased and many methods for character re-identification have been proposed, but a method with satisfactory performance has not yet been proposed. This is mainly due to the difficulties and challenges of re-identification of people operating on camera networks that do not share view. In particular, there are significant viewpoint, pose, and light changes between cameras that do not share their view, making it very difficult to identify people again.

Many existing approaches use patch-based methods to invariably re-identify people in terms of poses and perspectives [1-4]. This approach splits a pedestrian image into patches, extracts features from each patch, and then calculates the similarity between the two images using the similarity between the features of the patches. Because this approach is based on the local functionality of local patches instead of global capabilities, this approach has pose and viewpoint invariant properties. Nevertheless, this approach is not sufficiently time-invariant as it can still give false matching results when the images of different pedestrians observed at different points of view look similar.

An object of the present invention is to provide a method of object identification for view-invariant, global multi-object tracking for global multi-object tracking in cameras without visual field sharing.

The direction of pedestrians in the image is used to improve the viewpoint invariance of the patch-based method. If the human body is cylindrical, the proposed method estimates the angle of each patch according to the direction of the pedestrian. The patches then have cylindrical coordinates and the similarity between the patches is calculated not only based on the shape but also on the angle. The proposed method aims to show the viewpoint invariant property by matching the pedestrian image according to the exact coordinates of each local characteristic.

The human re-identification algorithm allows a local multi-target tracker (which works only for the same camera) to connect a local orbit of the same object in the local multi-target tracker to make one global trajectory, so that a global multi-target tracker (works for the camera network) Be sure to In order to accomplish this, the conventional method considers a still image data set but does not consider video data in a real-world scenario, and therefore aims to solve some problems that the conventional method cannot handle.

According to an embodiment of the present invention, an object re-identification method for global multi-object tracking comprises: extracting an image of one or more objects included in an image received from the outside; a group of pixels constituting the extracted object image Estimating an angular coordinate for each of the first patches; Comparing the angular coordinates of the second patches, which are a group of pixels constituting a previously stored object image, with the estimated angular coordinates; And re-identifying the one or more objects based on the comparison result.

The extracting may include deleting a background excluding the one or more objects from the frame of the image.

The estimating of the angular coordinates may include setting a moving direction of the object as a reference angle in the extracted image; And estimating an angular coordinate for each of the first patches according to the reference angle.

In the comparing, the angular coordinates of the first patch and the angular coordinates of all the second patches may be compared.

In parallel with the step of estimating the angular coordinate, the method may further include extracting color characteristics of the object for each of the first patches and performing a SIFT transform on the first patches.

The comparing may include comparing angular coordinates of the estimated first patches with angular coordinates of the second patches, comparing color characteristics of the extracted first patches with color characteristics of the second patches, and comparing the angular coordinates of the first patches. The method may include obtaining a similarity degree between the extracted object image and the previously stored object image based on a result of comparing the SIFT transform value of the patches with the SIFT transform value of the second patches.

The color characteristic extraction may include converting an RGB format into HSV and LAB formats for each of the first patches; And normalizing the element values of the converted format into a single characteristic vector.

The method may further include calculating a difference between the extracted color characteristic and the color characteristic of the second patches, and a difference between the SIFT transformed value and the SIFT transformed value of the second patches.

Averaging a difference between the value of the operation and the angle coordinates of the second patches and the estimated angle coordinates; And setting a minimum value of the averaged value as a difference value between the first patch and the pre-stored image.

The method may further include obtaining a similarity degree between the object image and the pre-stored image according to a difference value set for each of the first patches.

The method may further include obtaining a similarity degree between the extracted object image and the pre-stored image through an average of difference values set for each of the first patches.

The method may further include arranging the pre-stored image according to the obtained rank of similarity.

The method may further include applying the obtained similarity to a preset Gaussian distribution.

The preset Gaussian distribution may be characterized by a probability distribution according to similarity corresponding to the pre-stored object image and the non-pre-stored object image.

If it is determined that the extracted image matches the pre-stored image as a result of the application, the method may further include assigning an ID of the pre-stored image to the object of the extracted image.

If it is determined that the extracted image and the pre-stored image are inconsistent as a result of the application, the method may further include storing an image of the extracted object and assigning a new ID.

During the storage, the object of the extracted image of the object may be divided to store only the image of the object meeting the preset body ratio.

The present invention relates to a method for object re-identification for global multi-object tracking in a camera environment without visual field sharing, and to estimate the angular coordinates using the direction of movement of a person, and to compare the same, thereby recognizing a person who is robust to the change of viewpoint between cameras. In addition, it has the effect of reliable global multi-object tracking through the process of new detection and sample selection.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, included as part of the detailed description in order to provide a thorough understanding of the present invention, provide embodiments of the present invention and together with the description, describe the technical features of the present invention.

1 is a view showing the overall steps of the object re-identification method for global multi-object tracking according to an embodiment of the present invention.

2 shows the result of deleting the background, (a) the original image, (b) the modeled background, and (3) the result of the background deletion.

3 shows an example of pedestrian separation, grid cell (pixel) separation, patch angle estimation.

4 shows an example of angle estimation in a patch.

5 is a diagram illustrating the overall procedure of global multi-object tracking.

6 is a graph illustrating two Gaussian distribution functions using a training data set.

7 is a diagram illustrating an example of the results of body segmentation.

8 is a diagram illustrating an example of a VIPeR data data set.

9 shows performance in terms of nAUG.

10 and 11 show the arrangement of three cameras and the field of view of each camera.

In this study, we propose an invariant point of view method to re-identify people using the direction of pedestrians. The proposed method improves the viewpoint invariance of the patch-based method by using the pedestrian's direction in the image. If the human body is cylindrical, the proposed method estimates the angle of each patch according to the direction of the pedestrian. The patches then have cylindrical coordinates and the similarity between the patches is calculated not only based on the shape but also on the angle. The proposed method shows the view-invariant properties by matching the pedestrian image according to the exact coordinates of each local characteristic.

We also address the practical problem of applying human re-identification to multiple target tracking between cameras. In this work, the term global multi-target tracking refers to the tracking of moving objects in different camera fields of a camera network. In contrast, the term local multi-target tracking is defined as representing the tracking of a moving object in a single camera field of view. The human re-identification algorithm allows a local multi-target tracker (which works only for the same camera) to connect a local orbit of the same object in the local multi-target tracker to make one global trajectory, so that a global multi-target tracker (works for the camera network) Be sure to In order to do this, the conventional method considers a still image data set but does not consider video data in a real-world scenario, so some problems that the conventional method cannot solve are solved.

This involves two issues. The first is to select a pedestrian image that can identify the person again from the bounding boxes of the local tracker. This is called a sample selection problem. An appropriate sample selection method is needed because it is beneficial to use some very useful pedestrian images from all frames obtained as a local tracking result to shorten processing time. The second is to check whether the object observed in the camera network has appeared before. This problem is called new detection.

In this paper, we propose a framework to deal with two problems (multi-target tracking between cameras using human re-identification). For sampling, the reliability of each pedestrian image is calculated according to the ratio of three parts: head, upper body and lower body. We divide the three body parts in the pedestrian image using the asymmetric axis in [5], and select the pedestrian image with high reliability for human re-identification. For new detection we use two Gaussian distributions [5] calculated based on the similarity between false and exact matches. Given a match result of an object, if the probability of a bad match is higher than the probability of a correct match, the object is considered a new object.

To this end, the main contribution of this study is on two aspects: The first is to devise an invariant point-of-view method using the pedestrian's direction. The second is to propose a framework for dealing with global multi-target tracking that works on multiple separate cameras with cameras without vision sharing by adopting the proposed person's re-identification approach.

<Research on Human Reidentification>

Conventional human re-identification [4-11] uses an appearance-based approach using only images of objects taken from cameras. Other methods [12–14], on the other hand, use spatial and temporal relationships between cameras to improve performance as well as appearance. This method first learns the spatial and temporal relationships between cameras, and then uses the relationship information to predict where objects reappear (in the camera network). This method can distinguish objects that are difficult to distinguish from other objects when using the exterior alone, but there is a problem about how to learn spatial and temporal information between cameras.

Spatial and temporal relationship-based approach

Object tracking among existing methods using spatial and temporal information between cameras is used to model spatial and temporal relationships between cameras [12, 13]. Probability models between camera views are constructed to describe the probability that an object will appear in time and position. The brightness transfer function (BTF) can also be used to overcome non-overlapping pixel-to-pixel light variations. By clustering the time / end points of object tracking, start / end regions are found and visible and invisible links are estimated [12, 16]. In contrast, the relationship between cameras can be obtained without tracking an object [14]. First, the pattern over time is analyzed. Then we divide the camera view into regions and estimate the connections between regions based on pattern analysis.

Appearance-based approach

Appearance-based methods can be divided into (i) those who use the new feature descriptors, taking into account the special challenges of human re-identification, and (ii) those who deal with learning algorithms for distance measurement. Appearance-based methods basically use the similarity between feature descriptors extracted from the image of an object for human identification. A feature descriptor is calculated from the image to calculate the similarity between the two human images, and then the similarity between the feature descriptors is calculated using the appropriate distance metric. The feature descriptor-based approach focuses on constructing new feature descriptors that are useful for human re-identification, while the distance learning-based approach focuses on learning distance metrics more powerfully about changes in the environment between nonoverlapping camera views.

The assumption that pixels farther from the human axis of symmetry are more likely to belong to the background is used in [5]. First, the pixels in the foreground are compared to extract two horizontal asymmetric axes (dividing the human body into the head, upper body and lower body). The two vertical axes of symmetry, upper body and lower body, are then estimated. Next, when the feature is extracted from the image, the weight of the pixel closer to the axis of symmetry increases. For feature descriptors, weighted HSV color histograms [5], maximum stable color gamut (MSCR) [17] and repetitive high dimensional patches (RSHP) [5] are used. Body parts are detected using training data from [6]. Then compare appearances based on body parts. In addition to color bar graphs, the most widely used feature, various functions are used to re-identify a person. For example, a Gabor filter is used in [7] and a haar-like feature and dominant color descriptor are used in [8]. The patch matching method is used in [4]. The patch matching method splits the pedestrian image into partially overlapping patches and matches the features of the patches. The viability of each patch is further estimated and the patch matching results are weighted based on the effectiveness of the patch in [4]. Salience demonstrates the usefulness of patches for human identification and uses the k-nearest neighbor (KNN) or single-class support vector machine (OCSVM) to learn the importance of each patch. Use patch matching without any additional learning and additionally use the patch's angular coordinates for invariance to view point changes.

The distance metric learning approach trains the distance metric to make the distance within the class lower than the distance between classes instead of applying the same metric to all values in the property descriptor. Traditional metric learning methods include LMNN-R [9], PRDC [10], and RankSVM [11].

Discussion : Unlike existing methods, in order to ensure perspective invariant, the proposed method compares local feature descriptors considering local feature descriptors. For this purpose, the figures in the image are represented by a cylindrical model and the cylindrical coordinates of the local features are used to calculate the similarity between the local features.

In this study, we proposed a method for re-identification that is robust to viewpoint change. The angular coordinates of the local patch in the pedestrian image are used to match the local patch of other pedestrian images. For angular coordinate differences of matched patches in the image, the similarity of the patches is compared based on the position of the body. For this purpose, the pedestrian segmentation is performed first and the pedestrian direction is estimated. Then, the proposed method divides the pedestrian image into locally overlapping patches and estimates the angular coordinates for each patch. Finally, patch matching is performed by considering the difference between each coordinate to calculate the similarity between different pedestrian images.

The proposed method is also applied to the practical application of global multi-target tracking in cameras without visual sharing. In particular, two major challenges of sample selection and new detection have been properly addressed to realize a global multi-target tracking system.

The proportion of the body part is used to select the highly reliable sample calculated. The new detection is based on two Gaussian distributions in terms of similarity between pedestrian images. The two Gaussian distributions were constructed using the similarities of correct matching and wrong matching in the training set.

We have compared the proposed method with some existing methods to verify the importance of this study. Experiments performed on VIPeR data set (for human re-identification) and NLPR_MCT data set (global multi-target tracking). Experimental studies show that the proposed method can be significantly improved compared to the existing method in terms of accuracy.

In this study, LAB color histogram, HSV color histogram, and SIFT features were used to identify people. Future work will increase the accuracy by integrating various features with unique attributes into the function. You can also reduce the time complexity by using dimension reduction algorithms to reduce the dimension of the shape. Since the proposed method is based on a patch-based approach, other patch-based methods (eg distance metric learning) can be combined with the proposed method. In the case of multi-camera multi-target tracking, there is a problem with how to obtain a training set to estimate the threshold for new detection. Thus, methods for calculating more generalized thresholds and automatically gathering strong learning can be studied to improve the performance of new detections.

<Constant Person Reidentification Technology>

Overall procedure

The character re-identification system receives the pedestrian image to be identified, calculates the similarity between the input image and the image list composed of the image of the known person, and outputs the ranked list. The image of a person who needs identification is called a probe. A list of images consisting of known people's images is called a gallery.

To identify people again, the foreground must be separated from the pedestrian image. Next, divide the foreground into a patch set consisting of partially overlapping patches.

Then a feature descriptor is created for each patch and the angle of each patch is estimated. Feature descriptors and patch angles are used to calculate the similarity between pedestrian images. The similarity between the feature descriptors of the different pedestrian images and the angular difference of the different pedestrian images is calculated. Then, the patch is matched by considering similarities and differences.

Patches are matched to patches with high similarity of feature descriptors and small angle differences. Similarity between images is obtained by the average of the similarity of the matching patches. Finally, the ranking list is created by sorting in descending order based on similarity. The overall procedure is shown in FIG.

Pedestrian segmentation

In general, pedestrian images for person identification are rectangular and include a background. For accurate matching, the pixels that the pedestrian belongs to must be distinguished. Deep decompositional network (DDN) [18] can be used for this purpose. DDN uses deep learning to parse pedestrians from still images and to distinguish between people and the background. Despite the promising performance, we did not adopt it because of the complexity of the computation time. Pedestrians in the video can be distinguished using background subtraction and object detection. To do this, we use an average shift-based background removal method because of time efficiency [10]. 2 shows an example of a background deletion result in a video.

Orientation estimation

The proposed method uses the direction of the pedestrian image. Pedestrian directions are divided into eight classes, including 0, 45, 90, 135, 180, 225, 270 and 315 degrees. In the video we can get directions using two points in the pedestrian track. The direction of movement of the pedestrian is the direction of the image, assuming that a person will move forward. Orientation in still images can be obtained by conventional methods. Pedestrian images are categorized into eight directions using a histogram of slope (HoG) in [19]. The assumption throughout this work is that the still image is given the pedestrian's direction.

Division into grid

The pedestrian image is divided into a series of grid cells. Each grid cell is called a patch, and the grid cells partially overlap. In this task, the grid stage is 4 and the patch size is 10 x 10. Then, the angle from the orientation vector of the pedestrian is estimated, and a feature descriptor is constructed for each local patch.

Patch angle estimation

Angular coordinates are angles from the direction of movement of the estimated object. The reference direction is the direction of the pedestrian in the image. Each coordinate of each local patch can be estimated as follows. We know the coordinates of each pixel in the image. In addition, the horizontal distance between the pixel and the horizontal center of the pedestrian divided along the y-axis can be calculated. Therefore, the angle coordinates of each pixel can be calculated by replacing the distance from the center with the given circle equation as follows.

Claims (17)

Extracting an image of at least one object included in an image received from the outside; Estimating angular coordinates for each of the first patches that are a group of pixels constituting the extracted object image; Comparing the angular coordinates of the second patches, which are a group of pixels constituting a previously stored object image, with the estimated angular coordinates; And Re-identifying the one or more objects based on the comparison result; Object re-identification method for global multi-object tracking. The method of claim 1, The extracting step, Deleting a background excluding the at least one object from the frame forming the image, Object re-identification method for global multi-object tracking. The method of claim 1, Estimating the angular coordinates, Setting a moving direction of an object at a reference angle in the extracted image; And Estimating angular coordinates for each of the first patches according to the reference angle; Object re-identification method for global multi-object tracking. The method of claim 1, In the comparing step, Comparing the angular coordinates of one first patch and the angular coordinates of all the second patches, Object re-identification method for global multi-object tracking. The method of claim 1, In parallel with the step of estimating the angular coordinate, extracting the color characteristics of the object for each of the first patches, and performing a SIFT transform on the first patches, Object re-identification method for global multi-object tracking. The method of claim 5, The comparing step, Comparing the angular coordinates of the estimated first patches with the angular coordinates of the second patches, Comparing the color characteristics of the extracted first patches with the color characteristics of the second patches, The similarity between the image of the extracted object and the pre-stored object image is based on a result of comparing the first transform value of the first patch and the second transform value of the second patch to the SIFT transform. Including acquiring Object re-identification method for global multi-object tracking. The method of claim 5, The color characteristic extraction, Converting an RGB format into HSV and LAB format for each of the first patches; And Normalizing the element values of the transformed form and concatenating them into one characteristic vector; Object re-identification method for global multi-object tracking. The method of claim 5, The comparing step, The difference between the extracted color characteristic and the color characteristic of the second patches Calculating a difference between a first transform value of the SIFT transform and a second transform value of the SIFT transform on the second patches; Object re-identification method for global multi-object tracking. The method of claim 8, Averaging the difference between the color characteristics calculated in the calculating step, the difference between the first transform value and the second transform value, and the difference between the estimated angular coordinates and the angular coordinates of the second patches as a result of the comparison; And The method may further include setting a minimum value of the average value obtained in the averaging step as a difference value between the first patch and the pre-stored image. Object re-identification method for global multi-object tracking. The method of claim 9 The method may further include obtaining a similarity degree between the extracted object image and the pre-stored image according to a difference value set for each of the first patches. Object re-identification method for global multi-object tracking. The method of claim 10, The method may further include obtaining a similarity degree between the extracted object image and the pre-stored image through an average of difference values set for each of the first patches. Object re-identification method for global multi-object tracking. The method of claim 10, The method may further include arranging the pre-stored image according to the obtained rank of similarity. Object re-identification method for global multi-object tracking. The method of claim 10, The method may further include applying the obtained similarity to a preset Gaussian distribution. Object re-identification method for global multi-object tracking. The method of claim 13, The preset Gaussian distribution is, And a probability distribution according to similarity corresponding to the pre-stored object image and the non-stored object image. Object re-identification method for global multi-object tracking. The method of claim 13, If it is determined that the extracted image matches the pre-stored image as a result of the application, the object of the extracted image. The method may further include assigning an ID of a pre-stored image determined to match. Object re-identification method for global multi-object tracking. The method of claim 13, If it is determined that the extracted image and the pre-stored image are inconsistent, storing the extracted object image and assigning a new ID to the extracted object image; Object re-identification method for global multi-object tracking. The method of claim 16, In the storing, by dividing the object of the image of the extracted object to store only the image of the object meeting the preset body ratio, Object re-identification method for global multi-object tracking.

Images