WO2002011449A1

WO2002011449A1 - Representative presence for sensor-based security systems and other applications

Info

Publication number: WO2002011449A1
Application number: PCT/EP2001/008350
Authority: WO
Inventors: Daniel L. Pelletier
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2000-08-01
Filing date: 2001-07-18
Publication date: 2002-02-07

Abstract

A sensor-based system includes a number of sensors, each of which generates a sensor signal, and a processing system coupled to the sensors. The processing system processes the sensor signals received therefrom to generate a representative presence video signal suitable for presentation on a display. In an illustrative embodiment, one or more of the sensors each corresponds to a camera which generates a video signal. The processing system in such an embodiment is operative to compare at least a portion of each of the received video signals to stored information representing one or more patterns, to determine representations corresponding to one or more of the patterns detected within the video signals, and to generate the representative presence video signal such that it includes one or more of the representations. The representations in the output video signal may be indicative of normal or abnormal activity detected in the video signals generated by the cameras.

Description

Representative presence for sensor-based security systems and other applications

Field of the Invention

The present invention relates generally to the field of sensor signal processing, and more particularly to signal processing techniques involving representative presence and suitable for use in sensor-based security systems and other applications.

Background of the Invention

A number of systems known in the art process signals received from image sensors or other types of sensor devices in order to generate suitable output displays therefrom. For example, camera-based video signal processing systems may be configured to make use of a concept referred to as "representative presence. " Representative presence generally allows users participating in a video conference or other similar arrangement to be represented in a particular manner in a video signal displayed to remote participants. Each of the users may be provided with a certain amount of control over the manner in which they are represented in the displayed signal. An example of the conventional use of representative presence in a video conferencing application is described in U.S. Patent No. 5,347,306, entitled "Animated Electronic Meeting Place" and issued September 13, 1994 to Tohei Nitta. In this example, a network-based animated electronic meeting place includes a terminal for each participant on the network and local prestored animation graphics, with simulated participants in the form of animated characters or dolls being driven through sensing of each of the participants at their respective work stations, including their position, posture, gestures, audio content and optionally that persona which each individual wishes to be displayed at the animated electronic meeting place. A given user can thus choose how he or she represents himself or herself on a remote display screen through the use of a persona or character control. Although such representative presence techniques have been applied in video conferencing systems, a need exists for improved techniques for use in other video and sensor signal processing applications, such as security systems.

Conventional security systems generally lack an ability to provide effective summarization and communication of video signal information via representative presence. Examples of conventional security systems which involve video signal processing include U.S. Patent No. 3,767,851 entitled "Display Driver and System" and issued October 23, 1973 to Dale O. Wick, PCT Application No. WO 94/27408 entitled "Video Traffic Monitor for Retail Establishments and the Like" and published November 24, 1994, and European Patent Application No. EP 0 823 821 A2, entitled "System for Analyzing Movement Patterns" and published February 11, 1998.

Each of these conventional security systems processes one or more video signals or other types of sensor signals for presentation on a display. However, these and other existing techniques fail to make sufficiently effective use of representative presence. More particularly, these and other techniques fail to summarize diverse information present in the signals so as to facilitate understanding for one viewing and interpreting the displayed information.

Summary of the Invention

The present invention provides methods and apparatus for implementing representative presence in a manner particularly well suited for use in sensor-based security systems and other applications. The invention is generally applicable to any system which processes signals from multiple cameras or other types of sensors. In accordance with an aspect of the invention, a sensor-based system includes a number of sensors, each of which generates a sensor signal, and a processing system coupled to the sensors. For example, one or more of the sensors may each correspond to a camera which generates a sensor signal in the form of a video signal. The processing system processes the sensor signals received therefrom to generate a representative presence video signal suitable for presentation on a display. The processing system is operative to compare at least a portion of each of the received sensor signals to stored information representing one or more patterns, to determine representations corresponding to one or more of the patterns detected within the sensor signals, and to generate the representative presence video signal such that it includes one or more of the representations. By way of example, a camera-based system in accordance with an illustrative embodiment of the invention may be implemented in a security system application in which each of the cameras is associated with a particular area ofa facility being monitored. The representations in the output video signal may be indicative of normal or abnormal activity detected in the video signals generated by the cameras. More particularly, the processing system may be operative to generate a representative presence video signal which includes a number of representations, each of which may be indicative of the presence or absence of abnormal activity within a video signal generated by a corresponding one of the cameras in the illustrative embodiment. In accordance with another aspect of the invention, a characteristic of a given one of the representations within the representative presence video signal may be varied by the processing system to indicate a magnitude or other feature associated with a corresponding detected pattern.

The processing system of the present invention may be implemented as a centralized processing system, a distributed processing system, or as a combination of centralized and distributed processing elements.

Advantageously, the present invention permits the generation of representative presence video signals that efficiently characterize large amounts of video information derived from multiple cameras in a camera-based system, or sensor information derived from other types and arrangements of sensors, thereby facilitating understanding of the corresponding display. These and other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description.

Brief Description of the Drawings

FIGS. 1 A and IB are block diagrams showing exemplary implementations of an illustrative embodiment of a camera-based system with representative presence capabilities in accordance with the invention.

FIG. 2 is a flow diagram illustrating the operation of a video signal processing system in the camera-based systems of FIGS. 1 A and IB in accordance with the invention. FIG. 3 shows a number of example images illustrating a representative presence feature of the systems of FIGS. 1 A and IB in accordance with the invention.

Detailed Description of the Invention The present invention provides techniques for implementing representative presence in a security system or other camera-based system application. It should be noted that, although the illustrative embodiments described herein process video signals generated by cameras, the invention is more generally applicable to any sensor signal generated by any type of sensor device. For example, a system in accordance with the invention can process signals from motion detectors, fire or smoke sensors, alarms, etc. in addition to or in place of signals from cameras.

FIG. 1 A shows a camera-based system 100 in accordance with an illustrative embodiment of the invention. The system 100 includes a video signal processing system 102, a set of N cameras 104-1, 104-2, . . . 104-N, and a set of M displays 106-1, 106-2, . . . 106-M. The video signal processing system 102 includes one or more processors 110, memory 112, and a set of switching elements comprising switches and routers 114.

The cameras 104 may comprise, e.g., fixed- view cameras, zoom cameras, pan- tilt-zoom (PTZ) cameras, etc. All of the cameras of the system 100 need not be configured in the same manner, i.e., a given implementation of the system 100 may include a wide variety of different types of cameras.

The term "camera" as used herein is intended to include any type of image capture device which can be used in conjunction with the representative presence techniques of the invention. The term "sensor" is intended to include a camera or any other type of sensor device, such as a motion detector, fire or smoke sensor, alarm, etc.

The cameras 104 generate video signals that are supplied to the switches and routers 114 of the processing system 102. The video signals may be generated by the cameras 104 in an analog form, and subsequently converted to a digital form using elements of the processing system 102, such as analog-to-digital converters associated with the processor(s) 110. Alternatively, the cameras 104 may generate video signals in a digital form.

Each of the displays 106 may be configured to provide a presentation of one or more output video signals generated by the processing system 102. For example, each of the displays 106 may display various combinations of video signals generated by particular ones of the cameras 104, or a single video signal associated with a particular camera. One or more of the displays 106 may also provide a representative presence type of display generated in accordance with the techniques of the invention. An example of such a display will be described in greater detail below in conjunction with FIG. 3.

The processing system 102 receives the input video signals as generated by the cameras 104, processes the received signals in a manner to be described in greater detail below, and generates appropriate output video signals for presentation on the displays 106. One or more of the output video signals include representative presence type arrangements configured in accordance with the techniques of the invention.

It should be noted that certain elements or groups of elements of the camera- based system 100 of FIG. 1A may represent corresponding elements of an otherwise conventional computer, set of computers or security system, as well as portions pr combinations of these and other processing systems or devices. Moreover, some or all of the functions of individual elements of the processing system 102 may be combined into a single device. For example, one or more of these system elements may be implemented as an application specific integrated circuit (ASIC) or circuit card to be incorporated into a computer, television, set-top box or other processing device.

The one or more processors 110 of the processing system 102 are configured to execute system software to implement the representative presence techniques described herein in conjunction with FIGS. 2 and 3. The one or more processors 110 may thus represent, e.g., microprocessors, central processing units, microcontrollers, or any other data processing elements that may be implemented in the system, as well as portions or combinations of such elements. The memory 112 is configured to store at least a portion of the above-noted system software, and may comprise one or more electronic memories, disk- based memories, or other types of memories, as well as portions or combinations thereof. The memory 112 may be combined with the processor(s) 110 in a particular processing device, or may be implemented separately from the processor(s) 110. For example, a portion of the memory 112 may represent a memory of a microprocessor or ASIC that implements the processor(s) 110.

The processing system 102 of FIG. 1 A can be implemented as a centralized processing system, a distributed processing system, or as a combination of centralized and distributed processing elements.

FIG. IB shows one possible alternative implementation of the camera-based system 100 of FIG. 1 A. In this implementation, a camera-based system 100' includes a video signal processing system 102' having a combination of distributed and centralized processing elements. More particularly, the video signal processing system 102' includes processors 120-1, 120-2, 120-3, . . . 120-N, each associated with a corresponding one of the cameras 104. Each of the processors 120 may represent, e.g., a computer or other processing device, and may have a memory and one or more switching or routing functions associated therewith. The outputs of the processors 120 are supplied via a network 122 to a processing system 124, which processes the video signals, or signals generated from the video signals, in order to generate a representative presence output video signal.

The network 122 may comprise a local area network (LAN), a wide area network (WAN), or a global computer network such as the Internet, as well as portions or combinations of these and other networks. The processing system 124 may include a memory and one or more switching or routing functions, configured to operate ip a manner similar to that previously described in conjunction with the processing system 102 of FIG. 1A.

In one possible application of the systems 100 and 100' of FIGS. 1A and IB, respectively, the systems are configured as security systems, with each of the cameras 104 providing surveillance of a particular area of a building or other facility. Of course, the systems can also be used in numerous other applications.

The manner in which representative presence video signals are generated in accordance with the invention will now be described. The present invention provides techniques for utilizing representative presence as a data visualization method for representing large spaces of visual data. For example, in a security application involving systems 100 and 100' of FIGS. 1A and IB, respectively, there will typically be large amounts of video information received from the multiple cameras 104. Advantageously, representing certain types of presence or activity within the live video in abstract ways in accordance with the techniques of the invention make that information more accessible and easier to interpret than direct viewing of the live video itself.

The invention in the illustrative embodiment provides techniques for generating a representative presence video signal which includes abstract representations of people, objects, activities, etc. in a given surveillance space, such as that associated with a large security system in an airport, casino, building or other facility. The system may include a large number of cameras, e.g., on the order of tens or hundreds of cameras. The representations may be presented, e.g., in the form of a composite map-like view or other representation of the physical space, using one or more of the displays 106.

In accordance with one aspect of the invention, the abstract representations can be configured to highlight different patterns within the multiple input video signals supplied by the multiple cameras of the system. These detected patterns may be determined based on comparison, e.g., within the processing system 102 or 102' with a number of designated patterns previously stored in memory. For example, particular representations suitable for a representative presence type of display in accordance with the invention may be in the form of particular abstract representations indicating the presence of specified persons, objects or activities, a color change denoting the presence of more than a certain number of people within a given area, a highlighted trail denoting the presence ofa single person who moves abnormally through the space, or other representations denoting unusual relationships between persons within the space. It should be emphasized that these are only examples, and numerous other types of representations could be used.

FIG. 2 is a flow diagram illustrating the process of generating representative presence type output signals in accordance with the invention. This process in the illustrative embodiment may be implemented in the processing systems 102 and 102' of FIGS. 1 A and IB, respectively, using appropriately configured system software. In step 200 of FIG. 2, one or more video signals are received in the processing system 102 or 102' from the cameras 104. The processing system 102 or 102' in step 202 compares a frame or set of frames from each of the received input video signals to a number of previously-stored patterns corresponding to particular types of behavior of interest, as appropriate to the location under consideration. The processing system 102 or 102' then determines one or more representations appropriate for the detected patterns. Examples of such representations will be given in conjunction with FIG. 3. The processing system 102 or 102' in step 206 then generates an output video signal or set of output video signals that include the determined representations.

. In step 208, the output video signal or signals are displayed in accordance with specified user preferences, via one or more of the displays 106. For example, a user may specify the particular video outputs to be shown on certain displays, or the particular combination of live video and representative presence video outputs to be shown. FIG. 3 shows examples of the various video signals that may be processed or generated in the processing system 102 or 102' in accordance with the process shown in the FIG. 2 flow diagram. Element 302 in FIG. 3 is a set of live video frames 305-1 and 305-2 each corresponding to so-called normal activity in a given location under surveillance by the systems 100 or 100'. Element 304 is a set of live video frames 307-1 and 307-2 each corresponding to so-called abnormal activity in a given location under surveillance by the system 100 or 100'.

Each of the frames 305-1, 305-2, 307-1 and 307-2 may be associated with a different video signal generated by a corresponding one of the cameras 104. The classification of particular activity as "normal" or "abnormal" is performed by the processing system 102 or 102' in the manner previously described in conjunction with FIG. 2. More particularly, certain of the patterns stored in memory correspond to normal activity for an area monitored by a given camera, while other stored patterns correspond to abnormal activity for that area. By identifying frames corresponding to particular stored patterns, the processing system 102 or 102' can determine which input video signals include normal and abnormal activity.

In these examples, the processing system 102 or 102' generates a representative presence video signal including a set pf combined normal activity representations 310. A frame 312 of this representative presence video signal provides a placid undersea or fish tank view including as representations a number of different types of fish. The frame 312 is generated by the processing system 102 based on a determination that frames 305-1 and 305-2 include normal activity for their corresponding camera locations.

Each of the fish representations in the frame 312 may correspond to a particular camera or a particular room or other location in the facility being monitored. For example, one of the fish representations may represent the output of a camera which generated the video signal that includes frame 305-1, while another one of the fish representations may represent the output of a camera which generated the video signal that includes frame 305-2. The presence of a particular one of the fish representations in the video frame 312 thus provides a clear and easily understandable indication to a user that only normal activity has been detected in the corresponding video signal.

The processing system 102 or 102' may also generate a representative presence video signal including a set of combined abnormal activity representations 314. A frame 316 of this representative presence video signal includes representations 318 and 320, which correspond to types of abnormal activity detected in frames 307-1 and 307-2, respectively. More particularly, the shark representation 318 is indicative of the presence of an intruder in the video frame 307-1, and the volcano representation 320 is indicative of the presence of fire in the frame 307-2.

The processing system 102 or 102' in the FIG. 3 examples processes the set 310 of combined normal activity representations and the set 314 of combined abnormal activity representations, and generates therefrom an output representative presence video signal which includes a set 330 of combined normal and abnormal activity representations. A frame 332 of this signal as shown in FIG. 3 includes the fish representations of frame 312 indicative of the normal activity in frames 305-1 and 305-2, as well as the shark and volcano representations 318 and 320 of frame 316.

It should be noted that, in other embodiments, the frame 332 may be generated directly, without the intermediate step of separately generating the frames 312 and 316.

The output representative presence video signal including frame 332 may be presented to a user on a particular one of the displays 106 of the system 100 or 100', on a portion of one of the displays 106, or using a combination of the displays 106. The intermediate representative presence video signals corresponding to frames 312 and 316, as well as live video signals corresponding to input frames 305-1, 305-2, 307-1 and 307-2 may each be presented as an output video signal on one of the displays 106. The particular configuration of the signals presented on the displays 106 may be in accordance with pre- specified user preferences, or under direct real-time user control.

The abstract representations used in accordance with the present invention may be made scalable, e.g., the size, color or other characteristics of the representation can convey information regarding the magnitude of the importance associated with the detected pattern of activity within a given context, or other suitable feature associated with the detected pattern. It should be understood that the particular abstract representations shown in FIG. 3 are merely illustrative examples, and numerous other types of representations could be used, as appropriate for a given application.

Although particularly well suited for use in camera-based security system applications, the present invention may be used in other applications, such as video conferencing systems. In addition, as noted above, the invention can be used with a variety of other types of sensor signals in addition to or in place of the video signals used in the illustrative embodiments. Moreover, the particular types and arrangements of system elements shown in FIGS. 1 A and IB are by way of example only, and numerous alternative system configurations can be used to implement the representative presence techniques of the invention. For example, as previously noted, although the embodiment of FIG. 1 A shows centralized processing of the received video signals, other embodiments may implement a distributed processing arrangement in which a computer or other processing device is attached to each camera, in a manner similar to that shown in FIG. IB, or a combination of both centralized and distributed processing. The term "processing system" as used herein is intended to include such alternative arrangements. These and numerous other embodiments within the scope of the following claims will be apparent to those skilled in the art.

Claims

CLAIMS:

1. A sensor-based system (100, 100') comprising: a plurality of sensors (104), each generating a sensor signal; a processing system (102, 102') coupled to the sensors and receiving the sensor signals therefrom, the processing system being operative to compare at least a portion of each of at least a subset of the sensor signals to stored information representing one or more patterns, to determine representations (318, 320) corresponding to one or more of the patterns detected within the sensor signals, and to generate at least one output representative presence video signal (332) including one or more of the representations.

2. The sensor-based system of claim 1 further comprising at least one display

(106) for displaying at least a portion of the output representative presence video signal.

3. The sensor-based system of claim 1 wherein each of at least a subset of the sensors comprises a camera which generates a corresponding one of the sensor signals in the form of a video signal .

4. The sensor-based system of claim 1 wherein the system comprises a security system and each of the sensors is associated with a particular area ofa facility being monitored.

5. The sensor-based system of claim 1 wherein the processing system is further operative to generate a representative presence video signal comprising a plurality of representations, each of at least a subset of the representations being indicative of the absence of abnormal activity within a given one of the sensor signals generated by a corresponding one of the sensors.

6. The sensor-based system of claim 1 wherein the processing system is further operative to generate a representative presence video signal comprising a plurality of representations, each of at least a subset of the representations being indicative of the presence of abnormal activity within a given one of the sensor signals generated by a corresponding one of the sensors.

7. The sensor-based system of claim 1 wherein the processing system is further operative to generate a representative presence video signal comprising a plurality of representations, wherein the representative presence video signal comprises a composite video signal including a combination of one or more representations each indicative of the absence of abnormal activity within a given one of the sensor signals generated by a corresponding one of the sensors, and one or more representations each indicative of the presence of abnormal activity within another of the sensor signals generated by a corresponding one of the sensors.

8. The sensor-based system of claim 1 wherein a characteristic of a given one of the representations within the representative presence video signal is varied by the processing system to indicate a magnitude or other feature associated with a corresponding detected pattern.

9. The sensor-based system of claim 1 wherein the processing system is implemented at least in part as a centralized processing system.

10. The sensor-based system of claim 1 wherein the processing system is implemented at least in part as a distributed processing system.

11. A method for use in a sensor-based system (100, 100') comprising a plurality of sensors (104), each generating a sensor signal, and at least one display (106), the method comprising the steps of: comparing at least a portion of each of at least a subset of the sensor signals to stored information representing one or more patterns; determining representations (318, 320) corresponding to one or more of the patterns detected within the sensor signals; and generating at least one output representative presence video signal (332) including one or more of the representations, wherein at least a portion of the output representative presence video signal is suitable for presentation on the display.

12. The method of claim 11 wherein each of at least a subset of the serisors comprises a camera which generates a corresponding one of the sensor signals in the form of a video signal.

13. The method of claim 11 wherein the system comprises a security system and each of the sensors is associated with a particular area of a facility being monitored.

14. The method of claim 11 wherein the generating step further comprises generating a representative presence video signal comprising a plurality of representations, each of at least a subset of the representations being indicative of the absence of abnormal activity within a given one of the sensor signals generated by a corresponding one of the sensors.

15. The method of claim 11 the generating step further comprises generating a representative presence video signal comprising a plurality of representations, each of at least a subset of the representations being indicative of the presence of abnormal activity within a given one of the sensor signals generated by. a corresponding one of the sensors.

16. The method of claim 11 wherein the generating step further comprises generating a representative presence video signal comprising a plurality of representations, wherein the representative presence video signal comprises a composite video signal including a combination of one or more representations each indicative of the absence of abnormal activity within a given one of the sensor signals generated by a corresponding one of the sensors, and one or more representations each indicative of the presence of abnormal activity within another of the sensor signals generated by a corresponding one of the sensors.

17. The method of claim 11 further comprising the step of varying a characteristic ofa given one of the representations within the representative presence video signal to indicate a magnitude or other feature associated with a corresponding detected pattern.

18. The method of claim 11 wherein the comparing, determining and generating steps are implemented at least in part by a centralized processing system (102).

19. The method of claim 11 wherein the comparing, determining and generating steps are implemented at least in part by a distributed processing system (102').

20. An article of manufacture comprising a storage medium for storing one or more programs of a set of system software for controlling the operation of at least a portion ofa sensor-based system (100, 100'), the sensor-based system comprising a plurality of sensors (104), each generating a sensor signal, and at least one display (106), wherein the one or more programs when executed by a processing system of the sensor-based system implement the steps of: comparing at least a portion of each of at least a subset of the sensor signals to stored information representing one or more patterns; determining representations (318, 320) corresponding to one or more of the patterns detected within the sensor signals; and generating at least one output representative presence video signal (332) including one or more of the representations, wherein at least a portion of the output representative presence video signal is suitable for presentation on the display.