WO2006032799A1

WO2006032799A1 - Surveillance video indexing system

Info

Publication number: WO2006032799A1
Application number: PCT/FR2005/002372
Authority: WO
Inventors: Philippe Gouspillou
Original assignee: Video & Network Concept
Priority date: 2004-09-23
Filing date: 2005-09-23
Publication date: 2006-03-30
Also published as: FR2875629A1; FR2875629B1

Abstract

The invention relates to the field of video and, more specifically, to a method of characterising images from a video stream in order to find elements. The inventive method includes the following steps, namely: a first step comprising the acquisition of a video stream by an image sensor, a step comprising the digitisation of the acquired stream by a digitising module, a step comprising the characterisation of the digitised video stream by a processing unit, and a step comprising the encoding of the digitised video stream by an encoding unit. The invention is characterised in that: the characterisation and encoding steps are performed in parallel and in real time, the characterisation step produces a characterisation matrix for each of the images from the digitised video stream, and the method also includes a step comprising the timestamping of the stream processed during the processing step and encoded during the encoding step.

Description

VIDEO SURVEILLANCE INDEXING SYSTEM

The present invention relates to the field of video.

The present invention more particularly relates to a method for characterizing and indexing images of a video stream in order to facilitate searching for elements in a surveillance video stream.

Currently, even digital video surveillance systems are passive and involve slow human intervention in the use of images. For example, during a terrorist act or disappearance, investigations require the manual re-visualization of thousands of images, from surveillance cameras placed in the city or from all the video recordings that have been made in around the area where the desired action took place. These manipulations are slow and require a lot of staff and often lead to very long delays in obtaining results that allow to launch real actions in the field. Out in these tragic cases, it is the first hours that count. The more time passes and the likelihood of finding the traces of the aggressors or disappeared diminishes.

Recording systems used in video surveillance applications are already known in the state of the art. These systems allow the analysis of images, usually limited to motion detection (motion detection). When the detection result is active, the system saves the images. This solution makes it possible to limit the duration of the recordings in order to optimize the space of safeguard on the used medium, but has the disadvantage of being rudimentary and especially to allow no fine search in the images.

There are also systems that can analyze in a set of already recorded images an activity in an area of interest, for example, the disappearance of an object on a table. This solution requires significant processing times since the system must analyze a posteriori. In addition, the recording capacity is limited because of simultaneous searches and the bandwidth available on the network can also be limiting.

The prior art also knows, by US Pat. No. 6,646,676 (Paul DaGraca et al.), A monitoring and control system. This system performs a processing on an already compressed stream. This process is essentially based on motion detection in the context of an alarm trigger.

The prior art also knows, from US Pat. No. 6,721,454 (Richard Qian et al), a method for automatically extracting significant events from a video. This solution only detects the movement of objects within video sequences to determine events.

Also known by US Patent Application Nos. 6,424,370 and 5,969,755 (Jonathan D. Courtney) is a method for providing automatic video indexing of the motion of an object, based on the content. Moving objects in the video of a surveillance camera are detected in the video sequence by motion segmentation methods via a motion segmenter. Objects are tracked at through segmented data in an object tracker. A symbolic representation of the video is generated as annotated graphics describing the objects and their movement. A motion analyzer analyzes object tracking results and annotates graph movement with indexes describing multiple events. The graph is then classified using a rule-based classification scheme to identify events of interest such as appearance / disappearance, deposition / withdrawal, entry / exit, and motion / immobility of objects. Sequences of the video identified by spatio¬ temporal approach, event, and object-based queries are recalled to watch the desired video.

This process focuses on indexing video sequences. The mechanisms implemented evoke reference images, a graphic representation of the objects contained in the sequences. In addition, the characterizations of the scenes relevant to the criteria of appearance / disappearance, etc., are stored in a "frozen" manner (different from a continuous stream of elements characterizing a video stream) in a database. This solution realizes a motion segmentation of the video and creates a symbolic representation "meta-information" of the detected objects and their movement. These "meta-information" are stored in a database in the form of direct graphs annotated, then indexed. Such "meta-information" exists only for moving objects. Thus, it is clear that a characterization matrix can not be produced for each of the images of the stream and therefore a continuous stream of characterization matrices separate from the video stream is not created.

The prior art also knows, by the PCT patent application WO 98/21888 (Sarnoff Corporation), a method and an associated apparatus for representing completely video information so as to facilitate the indexing of this video information. This solution provides video preprocessing before performing an indexing step. A stream is cut into homogeneous scenes which are in turn cut into several levels of scene. All of these levels allow you to reconstruct the scene as a mosaic of key elements.

The prior art does not propose a solution allowing a light characterization (light compared to the high rate of video) of a video sequence by a flow of characterizing matrices.

Also known is the patent application US 2004/01611133 proposing a solution oriented behavioral analysis with a notion of self-learning behaviors. This solution integrates in particular the dissociation analysis of objects suggesting that an element is abandoned at a place (for example a suitcase deposited in an airport). The notion of image description which is very close to the functionalities of Media Asset Management tools (commercial name) is much more vague, poorly defined, and refers to complementary elements such as facial analysis. In this solution, behavioral analysis leads to: 1) record images, 2) characterize them to "illustrate" them. These events to be detected are predefined according to a multitude of parameters relating to the objects. This solution does not offer a continuous stream of characterization matrices associated with each image. This continuous flow provides the advantage of being able to define the events a posteriori.

The present invention has an inverse approach in providing a continuous flow of matrices. She describes images, in a systematic way without prejudging the behavior, which makes it possible later to analyze not on the images themselves but on sufficiently simplified simplified matrices.

It defines a description of images oriented towards the displacement of humans (and not vague objects) characterizing them by:

• their number;

• their "look" with 3 levels of characterization (top = head, middle = garment type shirt or sweater, bottom = garment down pants type);

• their displacement in the plane of the image - described by motion vectors in the image.

The characterization matrices thus obtained are "light" in terms of the amount of information and allow:

• the search for individuals by definition of the "look"

• a synthetic follow-up of the movements of the people (display mode) which gives the possibility of a behavioral analysis of type either visual by the operator, or automatic thanks to tools of behavioral analysis.

It is also known WOO3 / 067884 application proposing an image sequence processing solution for detecting / tracking objects and generating an alarm when the behavior of these tracked objects corresponds to a predefined behavior. A characterization of the objects tracked can be provided. However, this document does not propose a stream of characterization matrices timestamped and distinct from the video stream. Similarly, US6330025 is known in which high quality video recording is performed upon detecting a predetermined event. Possibly the operator can add annotations on the stream viewed. It is clear that a recording is only made when a predefined event is detected which goes to Against a characterization of each of the images as a stream of characterization matrices.

Similarly, the document WO03 / 028376 aims to capture images of people at control points. A keyframe and description data are associated with each captured video sequence. No "image" processing is performed to characterize each of them and to provide a flow of characterization matrices.

The present invention intends to overcome the drawbacks of the prior art by proposing a method for characterizing the image elements of a video stream by characterization matrices, these matrices constituting a data stream timestamped and parallel to the video stream.

The method according to the present invention responds particularly well to the search and the detection of elements inside a video stream and to the temporal constraints that they induce. Specifically, the invention applies both to a "real-time" detection by the detection of a matrix similar to that sought after than to the posterior detection by traversing the flow of characterizing matrices rather than the video stream, too much. heavy.

For this purpose, the invention relates in its most general sense to a method for characterizing images of a video stream for the search of elements, comprising:

A first step of acquiring a video stream by an image sensor; A step of digitizing said stream acquired by a scanning module,

A step of characterizing said digitized video stream by a processing unit;

A step of encoding said digitized video stream by an encoding unit, characterized in that

Said characterization and encoding steps are performed in parallel and in real time;

Said characterization step produces a characterization matrix of each of the images of said digitized video stream, and

In that it further comprises a step of timestamping of the processed flows resulting from the processing step and encoded resulting from the encoding step.

Preferably, said characterization matrix comprises:

• the percentage difference from the previous image; • percentage difference from a reference image;

• the number of detected mobile targets;

Characterization of said targets.

Advantageously, said target characterization comprises:

The position in the image of said targets,

The vector of displacement of said targets,

The dimension of said targets in relation to the overall image or to a reference,

The color characteristics of said targets,

The duration of presence of said target in the scene.

According to one embodiment, said method further comprises a step of capturing a fixed image zoomed on one of said characterized targets.

According to a variant, said method further comprises a step of associating said fixed image with the characterization matrix comprising said zoomed target. According to one embodiment, said capture is performed by an annex camera.

The invention also relates to a system for implementing the method comprising an image sensor, a frame memory, a scanning unit, a computing unit, an encoding unit comprising an encoding processor.

The invention also relates to a method of searching for elements in a video stream, characterized in that it implements the characterization method and in that it furthermore comprises:

A step of sending a request relating to the characteristics of said searched elements to the processing unit;

^• a step of comparing said characteristics with said characterization matrix generated during the characterization step.

The invention also relates to a method of searching for elements in a video stream, characterized in that it implements the characterization method and in that it further comprises: a step of transmitting said matrix of characterization generated during the characterization step at a remote server through a communication network;

A comparison step on said remote server of said characterization matrices with search criteria.

Preferably, said method further comprises:

A step of recovering the characteristics of a target in said matrices; and A step of displaying a spatial representation of the geographical positions of said targets in the scene on a display device.

The invention will be better understood by means of the description, given below purely for explanatory purposes, of an embodiment of the invention, with reference to the appended figures: FIG. 1 represents the architecture of the system according to the present invention;

FIG. 2 illustrates a simplified graphic representation of a scene analyzed by the present invention.

The invention relates to a method and a system for characterizing, in real time, the images of the CCTV video stream. This characterization is stored as a very small file in comparison with the weight of the images. These files are associated with the image streams by a time stamp and are either transmitted in parallel of the video stream or transmitted alone.

More specifically, the invention relates to a system for analyzing CCTV images to determine and retrieve key elements during search, in real time or a posteriori. The advantage of the method of the present invention is to allow a real-time analysis or a retrospective search on these files to search quickly, efficiently and simplified characteristic elements.

With reference to FIG. 1, the present invention comprises an image sensor (1) of the digital camera or video surveillance camera type. The sensor delivers analog or digital information containing a series of images (video signal for example). A step of digitizing the flow may be necessary if the sensor is analog. A frame memory makes it possible to store, temporarily (the processing time of the previous frame) and in real time, the frames generated by the image sensor. The image thus acquired and more precisely the frame stored in the frame memory, is transmitted to an encoding unit whose purpose is to compress the video signal and to obtain a bitstream (in which the images are indexed by the timestamping system. At the same time, the same image is transmitted to the processing unit whose purpose is to derive the essential characteristics of which the present text refers. The characteristic matrices thus obtained are also individually timestamped identically to the compressed images they characterize. The video stream and the matrix stream are then two separate data streams whose correspondence between matrix and image is ensured by the timestamping of these same streams.

The treatments according to the present invention are carried out, preferably in real time in order to maintain the best system performance (speed). Real-time processing means processing that lasts up to the time interval between two consecutive frames. Thus, each new frame is processed before receiving the next one.

The compressed bitstream and the sequence of characteristic matrices are then stored in a mass memory (memory unit) of application-dependent length: in general, several days to several months for video surveillance applications.

Finally, a network interface makes it possible to give access to the compressed bitstream or to the matrices characteristics. The external stations can also launch search queries to the processing unit, these searches being performed on the characteristic matrices. Once found the matrix corresponding to the search criteria (or reference image), it is possible thanks to the timestamp information and a set of index index to access the image or sequence of corresponding images.

According to the present invention, the characterization of the images is carried out in real time at the moment when they are acquired, recorded and possibly transmitted. The result of this characterization is a set of information, which we will call descriptive matrix, in the form of a file of small size compared to the size of the image, even compressed, which is associated with it. This association is achieved by a temporal marking method with the image that it characterizes. Some meta-data association systems of the MPEG-7 type (Motion Picture Expert Group) can be used to make this association.

This file is saved alongside the image. It can be exploited or transmitted alone, without the image that it characterizes.

In an application of the present invention, the search for elements may be subsequent to the recording of these data (matrices and video). The interest of the invention is to allow search through characteristic matrices and not images. Thus, it significantly simplifies the reprocessing and accelerates. We can deport the data analysis on servers after the transmission of the matrices. Finally, the bandwidth needs are greatly reduced given the small size of the files, which facilitates, moreover, the research on a large number of sources.

Another application of the present invention relates to real-time search. Characteristics of the searched element are sent to the processing server in real time. When creating the arrays, while the server receives the video stream from the sensor, the server compares the characteristics of the searched element with the matrices created in real time. If the characteristics match, an alarm can be raised.

The matrix of characterization is specific of the elements that one can be brought to monitor (human, car, object) that one calls "targets". The basic structure is composed as follows:

• percentage difference from the previous image; • percentage difference from a reference image;

• number of detected mobile targets;

• characterization of the targets.

The essential part of the matrix concerns the characterization of the targets since the search will be all the more refined if the characterizations have been well chosen and well realized. As characterization of the targets, we note: • the position in the image,

^• the displacement vector,

• the dimension related to the overall image or to a reference,

• the color characteristics: three "high - medium - low" zones, ^• the duration of presence of the target in the scene filmed and analyzed.

The three-zone color characteristics are particularly well suited to characterization and self-research, "high" corresponding to the head

(hair or hat), "average" in the coat and "low" in the pants.

The presence duration information of the target in the image makes it possible to carry out calculations on the duration of presence and possibly to establish rules following a too long presence in a passage zone, for example.

In a more advanced embodiment, the invention also makes it possible, in real time and from the coordinates of the targets, to make complementary photos to the overall video sequences. For this, one can use an image sensor, for example a digital camera separate from the camera, or use the same camera.

The photo can be made either by electronic zoom if the sensor of the camera used for the characterization is sufficient, or by an additional mobile camera that will focus on the target with a high magnification and will record the photo.

The fixed image thus recorded may also be associated with the characteristic matrix in order to allow a finer analysis. The "close-up" photos thus obtained are stored as objects with hypertext links allowing access to them.

In an embodiment illustrated in FIG. 2, a display device makes it possible to display a graphical representation of the elements analyzed in the scene by the present invention. Such a device can be realized by computer means (processor, memory and software for interpreting and transcribing the characteristic matrices of the targets) and a display screen.

FIG. 2 represents the display using simple shapes (round, square, etc.) of the geographic location of the targets (2) in the analyzed scene. An insert (3) informs of the number of targets present in the scene (four in this case) and the time of the analysis.

The device also allows the display of the instantaneous movement of the target with the help of arrows (6) but also that of the path (4) followed by the target (the historization can be done for example by saving the last positions of the target the target in the computer memory).

Finally, ancillary means, such as a computer mouse, to "point" a target on the screen and display the associated characteristics (5) to this target.

A very useful application of the present invention relates to the disappearance of a child in a city whose surveillance is provided by digital cameras or image recording servers implementing the method of the invention. To trace the child, it is sufficient to describe the characteristics of the child: • size, shape (human),

• color of the head, high garment, low garment,

• direction of movement.

The query can then be sent to the servers looking in the characteristic matrices if these criteria are found, and if they are not completely, what is the percentage of correspondence

(matching). It is then possible to visualize the images or sequences associated to perform a human validation for decision making. The invention is described in the foregoing by way of example. It is understood that the skilled person is able to realize different variants of the invention without departing from the scope of the patent.

Claims

A method of characterizing images of a video stream for searching for elements, comprising: a step of acquiring a video stream by an image sensor;

A step of digitizing said stream acquired by a scanning module,

A step of characterizing said digitized video stream by a processing unit;

A step of encoding said digitized video stream by an encoding unit providing an encoded video stream, characterized in that:

Said characterization step comprises a step of calculating a characterization matrix for each of the images of said digitized video stream and provides a stream of the matrices; Said matrix streams and encoded video stream are distinct; and

• in that it further comprises a step of time stamping the stream of the arrays and the encoded stream, said timestamp ensuring the correspondence between a matrix of said stream of the arrays and an image of said video stream.

2. Characterization method according to claim 1, characterized in that said characterization matrix comprises:

• the percentage difference from the previous image;

• percentage difference from a reference image; "The number of mobile targets detected; Characterization of said targets.

3. Characterization method according to claim 2, characterized in that said target characterization comprises:

The position in the image of said targets,

The vector of displacement of said targets,

The dimension of said targets in relation to the overall image or to a reference, the color characteristics of said targets,

The duration of presence of said target in the scene.

4. Characterization method according to one of claims 2 or 3, characterized in that it further comprises a step of capturing a fixed image zoomed on one of said characterized targets.

5. A method of characterization according to the preceding claim, characterized in that it further comprises a step of associating said fixed image with the characterization matrix comprising said zoomed target.

6. A method of characterization according to one of claims 4 or 5, characterized in that said capture is performed by a camera annex.

7. System for implementing the method according to any one of the preceding claims, comprising an image sensor for acquiring a video stream, a frame memory, a digitizing unit for digitizing said video stream, a computing unit, an encoding unit for encoding said digitized video stream and time stamping means, characterized in that: Said computing unit comprises means able to calculate a characterization matrix for each of the images of said digitized video stream and to provide a stream of the matrices distinct from said encoded video stream; Said time stamping means are able to timestamp said digitized video stream and said matrix stream to ensure correspondence between a matrix of said stream of the arrays and an image of said video stream.

8. A method of searching for elements in a video stream, characterized in that it implements the characterization method according to any one of claims 1 to 6 and in that it further comprises:

A step of comparing said characteristics with said characterization matrix generated during the characterization step.

9. A method of searching for elements in a video stream, characterized in that it implements the characterization method according to any one of claims 1 to 6 and in that it further comprises: step of transmitting said characterization matrix generated during the characterization step to a remote server through a communication network;

10. Element search method according to one of claims 8 and 9, characterized in that it further comprises: A step of recovering the characteristics of a target in said matrices; and

A step of displaying a spatial representation of the geographical positions of said targets in the scene on a display device.