JP7746114B2 - Image processing device, image processing method, image processing system, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, an image processing system, and a program.

Currently, there is an increasing number of systems that remotely control network-connected cameras (e.g., web cameras) to produce videos such as movies and music videos. Such systems use applications that can change the settings of each camera and control the camera's PTZ (pan, tilt, zoom) while the camera continues to capture images. The camera operator can use a device such as a joystick to change the settings and control the PTZ.

During video production, "cut-out" can occur when an object other than the performers (including people) that should appear in the footage, or an object unrelated to the footage (such as a cue card (a piece of information written by staff members to convey to performers)) unintentionally appears in the shooting range. In such cases, the camera operator will, for example, pan or tilt the camera to avoid the object, or use the zoom to prevent the object from appearing in the image. Patent Document 1 also discloses technology that allows the operator to set in advance the boundaries where "cut-out" will occur, and receive a notification when "cut-out" is about to occur.

JP 2018-125729 A

When operating a camera remotely, if an object (including a person) unrelated to the shooting suddenly enters the shooting range, resulting in a "cut-off" effect, the following problems may arise.
First, when operating remotely, it is difficult to instantly respond to the occurrence of a "cut-off" scene, and when one operator is operating multiple cameras, there is a possibility that the occurrence of a "cut-off" scene may be overlooked.
Furthermore, the technology of Patent Document 1 allows the operator to receive a notification when "cut-off" is likely to occur, but it is necessary to set a boundary where "cut-off" will occur in advance. Furthermore, in this document, the operator can receive a notification when the camera operation is about to exceed a predetermined boundary, but does not take into consideration the sudden intrusion of an object unrelated to the shooting range.

The present invention was made in consideration of the above-mentioned problems, and aims to provide technology for quickly detecting objects unrelated to photography that have entered the camera's shooting range and controlling the camera.

As one means for achieving the above object, the image processing device of the present invention has the following configuration: that is, it has an acquisition means for acquiring an image captured by an imaging means, a setting means for setting a first detection area in the image, which is an area extending a certain width inward from the outer frame of the image , and a second detection area, which is an area extending a certain width inward from the first detection area, a detection means for detecting an object that has entered at least one of the first detection area or the second detection area, a determination means for determining whether the object is an obstacle unrelated to image capture by the imaging means, and a control means for, if the determination means determines that the object detected in the first detection area is the obstacle, controlling the angle of view of the imaging means to remove the object from the range of image capture by the imaging means, wherein, if the object is detected in the first detection area after the control means has controlled the angle of view, the control means notifies the user, and if the object is detected in the second detection area after the control means has controlled the angle of view, the control means controls the angle of view of the imaging means to remove the object from the range of image capture by the imaging means.

This invention makes it possible to quickly detect objects that are not relevant to photography and that enter the camera's shooting range, and control the camera.

FIG. 1 shows an example of the overall configuration of an image processing system. FIG. 2 shows an example of the hardware configuration of the camera 102. FIG. 3 shows an example of the software functional configuration of the camera 102. FIG. 4 shows an example of the hardware configuration of the client device 103 . FIG. 5 shows an example of the software functional configuration of the client device 103. FIG. 6 is a flowchart of the process executed by the client device 103 according to the first embodiment. FIG. 7 is a diagram for explaining the imaging range according to the first embodiment. FIG. 8 is a flowchart of the process executed by the client device 103 according to the first embodiment. FIG. 9 explains the imaging range according to the third embodiment.

The following describes in detail an embodiment of the present invention, with reference to the accompanying drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in this embodiment are necessarily essential to the solution of the present invention.

First Embodiment
[System configuration]
1 is a diagram showing an example of the overall configuration of an image processing system according to the first embodiment. The image processing system in this embodiment includes a network 101, cameras 102-1 to 102-n (n>1), and a client device 103. In the following description, the cameras 102-1 to 102-n may be collectively referred to as cameras 102. The camera 102 and client 103 function as an image processing device.

Network 101 is a network that connects camera 102 and client device 103. Network 101 is realized, for example, by multiple routers, switches, cables, etc. that comply with communication standards such as ETHERNET (registered trademark). Note that network 101 may also be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc.

The camera 102 captures images through a lens (not shown) and acquires the captured images. The captured images are images captured within the imaging range (imaging range) of the camera 102. In this embodiment, the images include still images and/or videos. The camera 102 also manages information about the camera 102 (such as camera type information). In response to a request from the client device 103, the camera 102 can transmit the captured images and information about the camera 102 via the network 101. Furthermore, when the client device 103 is already connected, the camera 102 may actively transmit the captured images and information about the camera 102 acquired by the camera 102 to the client device 103.

The client device 103 is a general client terminal such as a personal computer (PC), tablet, or smartphone. The client device 103 can send requests to the camera 102 in response to user operations.

[Configuration of camera 102]
The hardware configuration and software functional configuration of the camera (image capture device) 102 according to this embodiment will be described with reference to FIGS.
FIG. 2 is a diagram showing an example of the hardware configuration of the camera 102 according to this embodiment.
The camera 102 includes an internal bus 201. The camera 102 is configured to include a CPU (Central Processing Unit) 202, a primary storage device 203, a secondary storage device 204, an imaging unit 205, a pan-tilt-zoom (PTZ) driving unit 206, and a network I/F 207, all of which are connected to the internal bus 201.

The CPU 202 controls the overall operation of the camera 102. That is, the CPU 202 can control the primary storage device 203, secondary storage device 204, imaging unit 205, PTZ drive unit 206, and network I/F 207.

The primary storage device 203 is a writable, high-speed storage device, such as RAM (Random Access Memory). For example, the OS (operating system), various programs, and various data are loaded into the primary storage device 203. The primary storage device 203 can also be used as a working area when the CPU 202 executes the OS, various programs, and the like.

The secondary storage device 204 is a non-volatile storage device such as a hard disk drive (HDD), flash memory, an SD card, etc. The secondary storage device 204 is used as a permanent storage area for the OS, various programs, various data, etc., and can also be used as a short-term storage area for various data, etc.
The secondary storage device 204 stores type information about the camera 102. In this embodiment, the type information stored is information indicating whether the camera 102 is a camera capable of pan, tilt, and zoom functions. For example, this information indicates that a camera capable of pan, tilt, and zoom is a PTZ camera, a camera capable of zoom only is a zoom camera, and a camera capable of neither pan, tilt, nor zoom is a fixed camera.

The imaging unit 205 is composed of, for example, an optical lens unit, an optical system that controls the aperture, zoom, focus, etc., and an imaging element that converts the light (image) introduced through the optical lens unit into an electrical signal. The imaging element typically uses a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The imaging unit 205 converts the subject light imaged by the lens included in the imaging unit 102 into digital data (electrical signals) using the imaging element and outputs the digital data.

The PTZ drive unit 206 controls the PTZ (pan, tilt, zoom) of the camera 102 by controlling motors and the like (not shown), and controls the angle of view of the camera 102, i.e., the imaging range. The angle of view is the imaging range (capture range) expressed as an angle. By controlling the pan, the angle of view of the camera 102 is moved in the horizontal direction. By controlling the tilt, the angle of view of the camera 102 is moved in the vertical direction. By controlling the zoom, the angle of view of the camera 102 is enlarged/reduced.
Furthermore, the PTZ driving unit 206 may be configured to control not only the PTZ control but also the settings and controls of the camera 102, such as focus and white balance. Furthermore, if the camera 102 is a video camera that is only capable of zooming or an interchangeable lens camera that does not have a zoom function, the camera 102 does not need to be equipped with the PTZ control unit 206.
The network I/F 207 is an interface for connecting to the network 101 .

Next, the software functional configuration executed by the CPU 202 of the camera 102 will be described. Figure 3 is a diagram showing an example of the software functional configuration of the camera 102 according to this embodiment. Processing according to the functional configuration shown in Figure 3 can be performed by the CPU 202 executing various programs stored in the primary storage device 203 and/or the secondary storage device 204.

The communication control unit 301 controls communication with the client device 103 via the network I/F 207 and the network 101. Specifically, the communication control unit 301 controls transmission and reception via the network I/F 207. For example, the communication control unit 301 can receive from the client device 103 an acquisition request for a captured image or a PTZ control request (hereinafter also referred to as a PTZ command) for the PTZ control unit 207 of the camera 102. Such requests may use a common communication protocol, such as HTTP (Hypertext Transfer Protocol).

The image processing unit 302 performs image processing (conversion processing) on the electrical signal generated by the imaging unit 205, and generates (acquires) the processed image data as a captured image. For example, the image processing unit 302 generates image data such as JPEG or video data such as H.264 as the captured image. This image processing can be performed in response to a request for acquiring the captured image received from the client device 103 via the communications control unit, for example. The image processing unit 302 may also be configured to perform this image processing in response to any other instruction.

The PTZ control unit 303 controls the PTZ control unit 207 based on the PTZ command received from the client device 103 via the communication control unit 301. Note that the PTZ control unit 303 not only physically controls the operation of the PTZ control unit 207, but may also perform digital PTZ, which performs pseudo-PTZ by cropping the captured image and changing the crop position.

[Configuration of client device 103]
Next, the hardware configuration and software functional configuration of the client device (control device) 103 according to this embodiment will be described with reference to FIGS.
FIG. 4 is a diagram showing an example of the hardware configuration of the client device 103 according to this embodiment.
The client device 103 includes an internal bus 401. The client device 103 is configured to include a CPU 402, a primary storage device 403, a secondary storage device 404, a user input/output I/F 405, and a network I/F 406, which are connected to the internal bus 401.

The CPU 402 controls the overall operation of the client device 103. That is, the CPU 402 can control the primary storage device 403, the secondary storage device 404, the user input/output I/F 405, and the network I/F 406.

The primary storage device 403 is a writable, high-speed storage device such as a RAM. For example, the OS, various programs, and various data are loaded into the primary storage device 403. The primary storage device 403 can also be used as a work area when the CPU 402 executes the OS, various programs, and the like.
The secondary storage device 404 is a non-volatile storage device such as a HDD, a flash memory, an SD card, etc. The secondary storage device 404 is used as a permanent storage area for the OS, various programs, various data, etc., and can also be used as a short-term storage area for various data, etc.

The user input/output I/F 405 is an I/F that allows the client device 103 to receive operations from a user, input user information, and output images and the like to the user. Examples of the user input/output I/F 405 include a display (display unit), a touch panel, a keyboard, a mouse, a microphone, a speaker, and the like. Here, the touch panel and the display may be configured as an integrated unit, such as a touch panel-equipped display (touch display) that can detect touch positions.
The network I/F 406 is an interface for connecting to the network 101 .

Next, we will explain the software functional configuration executed by the CPU 402 of the client device 103. Figure 5 is a diagram showing an example of the software functional configuration of the client device 103 according to this embodiment. Processing using the functional configuration shown in Figure 5 can be performed by the CPU 402 executing various programs stored in the primary storage device 403 and/or the secondary storage device 404.

The communication control unit 501 controls communication with the camera 102 via the network I/F 406 and the network 101. Specifically, the communication control unit 501 controls transmission and reception via the network I/F 406. For example, the communication control unit 501 can send an acquisition request for an image captured by the camera 102, or a PTZ control request (PTZ command) to the PTZ control unit 207 of the camera 102. Such requests may use a common communication protocol, such as HTTP.

The display control unit 502 controls the display of captured images acquired from the camera 102, buttons for controlling the camera 102, and the like on a display, which is one form of the user input/output I/F 405. The display control unit 502 can receive input via a mouse or keyboard, which serve as the user input/output I/F 405, or input via touch operations on a touch display, and can display various information.

The object detection unit 503 sets a detection area (detection region) for the captured image (shooting range) acquired via the communication control unit 501, and detects objects that invade the detection area (hereinafter also referred to as "invading objects"). The object detection unit 503 then determines whether the invading object is an object unrelated to the shooting (an object that should be excluded from the shooting range; hereinafter also referred to as "obstacle"). Note that in this embodiment, "objects" can also include people.

In this embodiment, the detection area is an area extending from the outer frame of the image/capture range (angle of view) captured by the camera 102 to a certain width inside. Note that the detection area may be set by the camera 102, and the object detection unit 503 may be configured to acquire information about the detection area via the communication control unit 501 and set it.

Whether an object in a captured image is an object to be detected by the object detection unit 503 can be determined by the object detection unit 503 or the camera 102. When the object detection unit 503 makes the determination, for example, the object detection unit 503 can analyze the captured image acquired via the communication control unit 501 using image processing to determine whether the object in the captured image is an object to be detected. When the camera 102 makes the determination, for example, the image processing unit 302 of the camera 102 analyzes the captured image using image processing to determine whether the object in the captured image is an object to be detected. The camera 102 can then notify the object detection unit 503 of information about the object to be detected via communications.

When the object detection unit 503 determines that the intruding object is an obstacle (an object unrelated to the image capture), it instructs the camera control unit 505 to control the PTZ of the camera 102. For example, the object detection unit 503 instructs the camera control unit 505 to move the intruding object out of the image capture range.

The timer unit 504 controls timing to make a predetermined judgment regarding an intruding object. The timer unit 504 is used by the object detection unit 503. For example, when the object detection unit 503 confirms, based on the timing of the timer unit 504, that a predetermined time has elapsed since the intruding object was present in the detection area, it determines that the intruding object is an obstacle.

The camera control unit 505 controls the camera 102 in response to an instruction from the object detection unit 503. For example, when the object detection unit 503 instructs the camera control unit 505 to move the intruding object out of the image capture range, the camera control unit 505 generates a PTZ command for controlling pan, tilt, and zoom of the camera 102 in response to the instruction. The camera control unit 505 then transmits the PTZ command to the camera 102 via the communication control unit 501. The control of pan, tilt, and zoom may be control of at least one of pan, tilt, and zoom. For example, it may be control for panning and tilting in a direction opposite to the direction of entry of the intruding object. Furthermore, the range of control of at least one of pan, tilt, and zoom may be set in advance, or may be automatically set so that the intruding object moves out of the image capture range.
For example, when the camera control unit 505 is instructed by the object detection unit 503 to restore PTZ control that has been performed once, the camera control unit 505 can generate a PTZ command to restore the PTZ settings in response to the instruction and send it to the camera 102 via the communication control unit 501.

The camera control unit 505 may be configured to control the camera 102 based on input from the user via the user input/output I/F 405.

[Processing flow]
6 shows a flowchart of the process executed by the client device 103 according to this embodiment. The client device 103 can repeatedly perform the process of this flowchart.

In S601, the object detection unit 503 detects (determines) whether an object has entered the detection area. In this embodiment, the detection area is an area extending from the outer frame of the image/capture range (angle of view) captured by the camera 102 to a certain width inside. If the presence of an object (intrusion) within the detection area is not detected within a certain period of time (No in S601), the processing ends. This certain period of time can be any period of time. On the other hand, if it is determined that an object is present within the detection area (Yes in S601), the processing proceeds to S602.

In S602, the object detection unit 503 sets a flag (object intrusion flag) indicating that an intruding object is present in the detection area (flag = 1). In response, the object detection unit 503 activates the timer unit 504 to start timing. In S603, the object detection unit 503 checks the timer unit 504 and determines whether a predetermined time has elapsed since the start of timing. If the predetermined time has not elapsed (No in S603), the timer unit 504 continues timing. If the predetermined time has elapsed (Yes in S603), the processing proceeds to S604.

In S604, the object detection unit 503 determines whether the object detected in S601 (intruding object) is still present within the detection area after a predetermined time has elapsed. If it is determined that the intruding object is not present within the detection area (No in S604), the processing proceeds to S605, where the object detection unit 503 clears the object intrusion flag (flag = 0), cancels the timer unit 504, and ends the processing.

On the other hand, if it is determined that the intruding object is still present within the detection area after the predetermined time has elapsed (Yes in S604), the object detection unit 503 determines that the intruding object is an obstacle (an object unrelated to the image capture). In this case, the object detection unit 503 instructs the camera 102 to move the intruding object out of the image capture range, and the process proceeds to S606. In S606, the camera control unit 505 performs PTZ control (angle of view control) on the camera 102 so that the intruding object is out of the image capture range. For example, the camera control unit 505 transmits a PTZ command indicating control of at least one of pan, tilt, and zoom to the camera 102 via the communication control unit 501 in order to move the detection area out of the image capture range. Upon receiving the command, the PTZ control unit 303 of the camera 102 performs digital PTZ control in accordance with the command and adjusts the angle of view of the camera 102 so that the intruding object is out of the image capture range.
Furthermore, in S606, the object detection unit 503 stops the operation of the timer unit 504 (cancels the timer), and the process proceeds to S607.

In S607, the object detection unit 503 again determines (detects) whether the object detected in S601 is present within the detection area. If no intruding object is detected within the detection area within a certain period of time (No in S607), the intruding object flag is cleared (flag = 0) in S608, and processing proceeds to S609. This certain period of time may be any period of time. In S609, the camera control unit 505 controls the camera 102 to return the PTZ control of S606 to its original state. For example, the camera control unit 505 transmits a PTZ command indicating return of PTZ control to the camera 102 via the communication control unit 501. Upon receiving this command, the PTZ control unit 303 of the camera 102 performs PTZ control in response to the command to return the camera 102 to its original shooting range. Then, processing ends. On the other hand, if it is determined that an intruding object is present within the detection area within the certain period of time (Yes in S607), the determination continues.

In the above explanation, in S606, the camera 102 performed digital PTZ control in response to the PTZ command sent by the object detection unit 503. However, this is not limited to this; for example, optical PTZ control may be performed. In this case, the object detection unit 503 cannot determine from the captured image whether an intruding object is present in the detection area (S607). Therefore, the camera control unit 505 may transmit a PTZ command to the camera 102 indicating that the PTZ control should be returned to the original setting after a certain time has elapsed since S606.

In the above description, in S601, the object detection unit 503 detects (determines) whether an object has entered the detection area, but the direction of entry of the object may also be taken into consideration. For example, the object detection unit 503 may determine that an object has entered the detection area from outside, but may not determine that an object has entered the detection area from inside toward outside. In other words, in this case, the object detection unit 503 may perform the process of S601 for an object that has entered the detection area from outside, but may not perform the process of S601 for an object that has entered the detection area from inside toward outside.
In addition, instead of the processing of S601 to S604, when the object detection unit 503 detects that an intruding object has entered the detection area from the outside to the inside, the object may be determined to be an obstacle (an object unrelated to photography) and the process may proceed to S606.

In the above description, in steps S602 to S604, the object detection unit 503 determines whether the intruding object is an obstacle (an object unrelated to filming) based on whether the intruding object is still present in the detection area after a predetermined time has elapsed. However, the determination method is not limited to this. For example, the object detection unit 503 may determine whether the intruding object is an obstacle based on pre-registered facial information or the color of the performer's costume. That is, the object detection unit 503 determines whether the intruding object is a pre-registered object, and if it determines that the intruding object is not a registered object, it determines that the intruding object is an obstacle, and the client device 103 can perform the processing from step S606 onwards. Furthermore, if the object detection unit 503 determines that the intruding object is not a registered object and a predetermined time has elapsed since the intrusion, it may determine that the intruding object is an obstacle, and the client device 103 may perform the processing from step S606 onwards.

In addition, in this embodiment, the object detection unit 503 analyzes the captured image to determine whether or not an object is a target for detection, but this is not limited to this. For example, a person involved in filming (such as a performer) may wear (attach) a sensor or the like as a registered object, and the object detection unit 503 may determine that an intruding object is an obstacle by receiving (detecting) a signal (sensor signal) from the sensor. In other words, when the object detection unit 503 receives a signal from a sensor attached to the intruding object, it can determine that the intruding object is not an obstacle even if a predetermined time has passed since the reception. Conversely, when an intruding object enters the detection area, if a predetermined time has passed without receiving a sensor signal, the object detection unit 503 determines that the intruding object is an obstacle, and the client device 103 can perform processing from step S606 onwards.

Next, referring to FIG. 7, the imaging range before and after PTZ control according to this embodiment will be described. FIG. 7 is a diagram for explaining the imaging range (angle of view) according to this embodiment. In FIG. 7, imaging range 701 indicates the imaging range (angle of view) captured by camera 102. Imaging range 701 may be the imaging range at the start of imaging. When object 704 as an intruding object enters detection area (shaded area) 702 (Yes in S601), the object detection unit 503 determines whether object 704 will remain in detection area 702 for a predetermined time (S602, S603). Thereafter, when the predetermined time has elapsed while object 704 remains in detection area 702 (Yes in S603), the camera control unit 505 determines object 701 as an obstacle and sends a PTZ command to camera 102 to remove detection area 702 from imaging range 701 (S606). In response to this, the camera 102 performs PTZ control, and the shooting range 701 is changed to the shooting range 703 after PTZ control.

As described above, according to this embodiment, if an object detected as having entered the detection area satisfies a predetermined condition, the client device 103 determines that the object is an object unrelated to the image capture (an obstacle), and controls the camera 102 to remove the object from the image capture range. This enables the client device 103 to automatically detect obstacles and automatically control the camera. Furthermore, as shown in FIG. 1, this image processing system is configured to include multiple cameras 102-1 to 102-n, and the client device 103 can perform the above-described control for each camera. Note that the client device 103 may also be configured to perform the above-described processing only on images to be used for broadcast.

Note that while this embodiment has been described using an example of control by the client device 103, the same explanation can be applied to control by the camera 102. For example, if the camera 102 detects an object entering a detection area and remains in the detection area for a predetermined period of time, the camera 102 can determine that the object is an obstacle and control itself to remove the object from the shooting range.

Second Embodiment
Next, a second embodiment will be described. In this embodiment, after an object entering a detection area, which is an area outside the imaging range captured by the electronic PTZ within the imaging range captured by the optical PTZ, is determined to be an obstacle, a camera operator is notified of the presence of the object. The camera operator may be the user/operator of the client device 103. Below, differences from the first embodiment will be described, and a description of configurations/features common to the first embodiment will be omitted.

Figure 8 shows a flowchart of processing executed by the client device 103 according to this embodiment. The client device 103 can repeat the processing of this flowchart. In this embodiment, the detection area is defined as an area outside the capture range (captured image) cut out by the electronic PTZ from the capture range (captured image) of the optical PTZ captured by the camera 102, but within the capture range (captured image) of the optical PTZ. In other words, the detection area is defined as an area in the captured image extending a certain width inward from the edge of the image. A partial image excluding the detection area is cut out from the image, and the partial image is transmitted from the client device 103 to an external device. If an object determined to be an obstacle is detected in the detection area, the client device 102 notifies the camera operator of the presence of an intruding object. In this embodiment, when a partial image is cut out by the electronic PTZ from the capture image of the optical PTZ capture range in the camera 102, the captured image of the optical PTZ capture range and the partial image are transmitted to the client device 103. The client device 103 then receives (acquires) the captured image and the partial image, and sets the area of the captured image other than the partial image as a detection area. If the client device 103 subsequently detects an object determined to be an obstacle in the detection area of the captured image (a captured image of the optical PTZ's capture range), it notifies the camera operator of the presence of an intruding object. The processing of the client device 103 in this embodiment will now be described with reference to the processing flow in Figure 8. Processes similar to those in the flowchart shown in Figure 6 are assigned the same reference numerals and will not be described again.

8 , if it is determined that an object is still present in the detection area (Yes in S604) after a predetermined time has elapsed since the detection of an intrusion of an object into the detection area (S601), the process proceeds to S801. In S801, the object detection unit 503 notifies the camera operator of the presence of the intruding object, i.e., that the intruding object is present as an obstacle in the detection area, via the user input/output I/F 405. The notification can be made via a display (e.g., displaying a warning screen) or a speaker (e.g., emitting a warning sound) as the user input/output I/F 405.
Furthermore, in S801, the object detection unit 503 simultaneously stops the operation of the timer unit 504 (cancels the timer), and the process proceeds to S607.

As described above, according to this embodiment, with PTZ control already in place, the client device 103 notifies the camera operator of the presence of an intruding object if the object has entered the detection area and remains there for a predetermined period of time. This allows the camera operator to recognize the presence of an intruding object in the detection area before it enters the shooting range.

Third Embodiment
Next, a third embodiment will be described. In this embodiment, as in the first embodiment, two detection areas (an outer detection area and an inner detection area) are provided for detecting an intruding object after PTZ control is performed when the intruding object is determined to be an obstacle. If an object intrudes into the outer detection area, the camera operator is notified of the presence of the intruding object. If the object intrudes into the inner detection area, the object is determined to be an obstacle and digital PTZ control is performed.

The shooting range before and after PTZ control according to this embodiment will be explained with reference to Figures 9(a) and 9(b). Figures 9(a) and 9(b) are diagrams for explaining the shooting range (angle of view) according to this embodiment.

In Figure 9, the shooting range 901 indicates the shooting range (angle of view) captured by the camera 102. The shooting range 901 may be the shooting range at the start of shooting. The first detection area (shaded area) 902, which is the outer detection area, and the second detection area (diagonal line area) 905, which is the inner detection area, are detection areas for detecting an intruding object. In this example, when an intruding object 904 enters the first detection area 902, the camera 102 performs PTZ control (S606 in Figure 6), and the shooting range of the camera 102 is changed to the shooting range 703 after PTZ control.

Figure 9(a) shows that object 904 is remaining in first detection area 902. Here, if the object detection unit 503 of the client device 103 determines that object 904 has remained in first detection area 902 for a predetermined time, it performs processing according to the second embodiment (see Figure 8). That is, if it determines that object 904 has remained in first detection area 902 for a predetermined time (Yes in S604 in Figure 8), the client device 103 notifies the camera operator of the presence of an intruding object (S801 in Figure 8).

Also, Figure 9(b) shows that object 904 has entered the second detection area 905. At this time, the client device 103 performs processing according to the first embodiment (see Figure 6). That is, when a predetermined time has passed since object 904 entered the second detection area 905 (Yes in S603 in Figure 6), the object detection unit 503 of the client device 103 sends a PTZ command to the camera 102 to remove the object 904 from the shooting range (S606). In response, the camera 102 performs PTZ control, and the shooting range is changed to the shooting range after PTZ control 906.

As described above, according to this embodiment, two detection areas are provided: an outer detection area and an inner detection area. Then, with PTZ control already in place, the client device 103 notifies the camera operator of the presence of an object that has entered the outer detection area and remains there for a predetermined period of time. Furthermore, if an object enters the inner detection area and meets predetermined conditions, the object is determined to be an object unrelated to photography (an obstacle), and the camera 102 is controlled to remove the object from the photography range. This enables the client device 103 to automatically detect and notify of obstacles and automatically control the camera.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

101: Network, 102: Camera, 103: Client device

Claims (11)

an acquisition means for acquiring an image captured by the imaging means;
a setting means for setting a first detection area in the image, which is an area extending from an outer frame of the image to a certain width inward, and a second detection area in the image, which is an area extending from the first detection area to a certain width inward;
a detection means for detecting an object that has entered at least one of the first detection area or the second detection area;
a determination means for determining whether the object is an obstacle that is not relevant to the image capturing by the image capturing means;
When the object detected in the first detection area is determined to be the obstacle by the determination means,
a control unit that controls the angle of view of the imaging unit so as to remove the object from the imaging range of the imaging unit,
after the control means has controlled the angle of view, if the object is detected from the first detection area, a notification process is executed to notify a user;
an image processing device comprising: after the control means has controlled the angle of view, if the object is detected from the second detection area, controlling the angle of view of the imaging means so as to remove the object from the imaging range of the imaging means. An image processing device according to claim 1, characterized in that the determination means starts timing when the object enters the first detection area, and determines that the object is an obstacle when it detects that the object has remained in the first detection area for a predetermined period of time. 2. The image processing device according to claim 1, wherein the determining means detects the direction of the object's intrusion into the first detection area, and determines that the object is an obstacle when it detects that the object has intruded from outside to inside the first detection area. An image processing device according to claim 1, characterized in that the determination means determines whether the object is a pre-registered object, and if it determines that the object is not a pre-registered object, determines that the object is an obstacle. The image processing device described in claim 4, characterized in that a sensor is attached to the registered object, and the determination means determines whether the object is the registered object based on whether a signal is received from the sensor. An image processing device according to any one of claims 1 to 5, characterized in that the control means controls the angle of view of the imaging means using digital PTZ (pan-tilt-zoom) or optical PTZ. An image processing device according to any one of claims 1 to 6, characterized in that if the object is not detected from the first detection area and the second detection area for a predetermined period of time after the angle of view is controlled by the control means, the control means controls the shooting range to the angle of view before the angle of view was controlled by the control means. An image processing device according to any one of claims 1 to 7, characterized in that the image processing device has the imaging means. An image processing system comprising a plurality of imaging devices each having the imaging means and an image processing device according to any one of claims 1 to 7. an acquisition step of acquiring an image captured by an imaging means;
a setting step of setting a first detection area in the image, which is an area extending from an outer frame of the image to a certain width inward, and a second detection area in the image, which is an area extending from the first detection area to a certain width inward;
a detection step of detecting an object that has entered at least one of the first detection area or the second detection area;
a determination step of determining whether the object is an obstacle that is not relevant to the image capturing by the image capturing means;
a control step of controlling an angle of view of the imaging means so as to remove the object from a photographing range of the imaging means when the object detected in the first detection area is determined to be the obstacle in the determination step,
after the angle of view is controlled in the control step, if the object is detected from the first detection area, a notification process is executed to notify a user;
an image processing method characterized in that, after the control of the angle of view has been performed in the control step, if the object is detected from the second detection area, the angle of view of the imaging means is controlled so as to remove the object from the shooting range of the imaging means. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 7 .