WO2015193995A1 - Projection picture display device, projection picture display method, and operation detection device - Google Patents

Abstract

The present invention provides a projection picture display device capable of identifying an operating user and switching processing contents in accordance with the user. The projection picture display device is provided with: an image-capturing unit (102) that captures an image of a picture projection area on a projection surface; an operation detection unit (105) that detects the content of an operation performed by the user with respect to picture display on the basis of the image captured by the image-capturing unit; a user identification unit (107) that identifies the user who performed the operation on the basis of the image captured by the image-capturing unit; a picture processing unit (154) that processes a picture, to be displayed on the projection surface, in accordance with the content of the operation by the user and a user identification result; and a picture projection unit (153) that projects the picture processed by the picture processing unit onto the projection surface.

Description

Projection-type image display device, image display method, and operation detection device

The present invention relates to a projection display apparatus and an image display method for detecting a user operation and displaying an image in response thereto, and an operation detection apparatus used therefor.

As a user operation input on the projection surface of a projection display apparatus (projector), a technique for detecting the operation content by photographing the user's operation unit (finger) without using a special device such as a touch sensor is proposed. Has been.

Patent Document 1 discloses a region of the operator's specific part based on a unit that causes the imaging unit to image the operator in a state illuminated by the illumination unit, and an operator's image data obtained by the imaging unit. Means for detecting, means for extracting a shadow part from the detected region of the specific part of the operator, and detecting and detecting a plurality of line segments in which the edges form a straight line from the extracted shadow part There is described an operation detection device comprising: means for detecting a point at which the line segments intersect each other at an acute angle, and detecting the intersection as a pointing position in the region of the specific part of the operator.

Patent Document 2 discloses a projection unit that projects an image on a screen, an imaging unit that captures at least a region including the image projected on the screen, and an image captured by the imaging unit. A real image detection unit for detecting a real image of a predetermined object moving upward; a shadow detection unit for detecting a shadow of the predetermined object generated by projection light from the projection unit from an image captured by the imaging unit; When the distance between the corresponding point of the real image of the object and the shadow is equal to or less than a predetermined threshold, the contact determination unit determines that the predetermined object is in contact with the screen, and the contact determination unit makes contact And a coordinate determining unit that outputs the coordinates of the predetermined object as a pointing position with respect to the image when it is determined.

JP 2008-59283 A JP 2011-180712 A

In a conventional projection display apparatus, user operations can be detected from captured images to control image display. However, when multiple users operate, the same processing is performed regardless of which user operates. It becomes. Therefore, when a plurality of users perform different operations at the same time, a plurality of processes may compete and normal processing for each user may be difficult. Alternatively, even when a plurality of users are switched and operated, stable processing may be difficult if different processing is frequently performed for each user.

In Patent Document 1, a shadow portion is extracted from the image data of the operator, and a point where a shadow line segment intersects at an acute angle is detected as a pointing position. Further, an acute bisector formed by the line segments is obtained, a position where the bisector intersects with the contour of the specific part of the operator is detected, and the pointing operation is performed by the operator based on the detected position. It is determined whether it was performed or performed by a person other than the operator. However, it is not considered to detect who the operator is and switch the processing contents according to the operator. That is, when a plurality of operators operate at the same time or change, it is not possible to perform video display processing while distinguishing the operators.

In Patent Document 2, when the distance between the real image of the predetermined object (finger) and the corresponding point of the shadow is equal to or smaller than a predetermined threshold value, it is determined that the predetermined object is in contact with the screen. Also in this case, it is not considered to detect who the operator is and switch the processing contents according to the operator.

An object of the present invention is to provide a projection video display device and a video display method capable of identifying a user to be operated and switching processing contents according to the user, and an operation detection device used therefor.

The present invention includes a plurality of means to solve the above problems. For example, in a projection display apparatus that projects and displays an image on a projection plane, an imaging unit that captures an image projection area on the projection plane and an image display based on an image captured by the imaging unit. An operation detection unit that detects the operation content of the user, a user identification unit that identifies the user who performed the operation based on the image captured by the imaging unit, and a projection plane according to the user operation content and the user identification result An image processing unit that processes an image displayed on the image processing unit, and an image projection unit that projects the image processed by the image processing unit onto a projection surface.

Alternatively, in an image display method for projecting and displaying an image on a projection surface, an imaging step for capturing an image projection area on the projection surface, and detecting a user operation content for image display based on the image captured by the imaging step Based on the operation detection step, the image captured in the imaging step, a user identification step for identifying the user who performed the operation, and processing of the video displayed on the projection surface according to the user operation content and the user identification result And a video processing step to be performed.

According to the present invention, the user of the operation is identified, and the processing content is switched according to the user, so that the processing of the projection display apparatus is stabilized even when a plurality of users operate simultaneously or by switching. User convenience is improved.

FIG. 1 is a block diagram illustrating a configuration of a projection display apparatus (Example 1). The figure which shows the external appearance of a projection type video display apparatus, and a user's operation state. The figure which shows the difference in the shape of the shadow by the contact / non-contact of a finger | toe and an operation surface. The figure which shows the relationship between the distance of a finger | toe and an operation surface, and the shape of a shadow. The figure which shows the process which detects the contact point of the finger | toe and operation surface in the operation detection part. The figure which shows the user registration process in the user management part. The figure which shows the example of the parameter showing the characteristic of a user's hand. The figure which shows the user registration processing flow by manual. The figure which shows the user identification processing flow in the user identification part 109. FIG. The figure which shows the example which projects a different image | video with respect to a different user. The figure which shows the example which displays an image | video with a different display format with respect to a different user. The figure which shows the example which displays a different menu with respect to a different user. The figure which shows the example which performs a different display process with respect to a different user. The figure which shows the example which restrict | limits operation according to a user's attribute. The figure which shows the other example which restrict | limits operation according to a user's attribute. FIG. 10 is a diagram illustrating an example of user data managed by a user management unit 106 (second embodiment). The figure which shows the user registration processing flow by manual. FIG. 10 is a diagram illustrating an example of user data managed by a user management unit 106 (third embodiment). FIG. 10 is a diagram for explaining processing for detecting a contact point between a pen and an operation surface (Example 4). The figure which shows the example of the user data which the user management part 106 manages.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

In the first embodiment, a method of detecting a user operation while imaging a user's hand with a camera and identifying the user to operate will be described. Therefore, the features of each user's hand are acquired in advance, a user ID and attribute are assigned to each user, and video display processing is performed according to the operation according to the user ID and attribute of each user. Here, the user ID is information for identifying the user, and the attribute is information that classifies the user according to the operation authority and assigns the user according to the operation authority.

FIG. 1 is a block diagram illustrating a configuration of a projection display apparatus (hereinafter also referred to as an image display apparatus) according to the first embodiment. The detection function unit 101 is a function unit that detects a user's operation. The camera (imaging unit) 102, two illuminations 103 and 104, an operation detection unit 105, a user management unit 106, a user identification unit 107, a communication unit 108, and a control unit. 109. The display function unit 151 is a function unit that displays an image in accordance with a user operation, and includes a communication unit 152, an image projection unit 153, and a control unit 154. The detection result data 121 is transferred from the detection function unit 101 to the display function unit 151.

First, each component of the detection function unit 101 will be described.
The camera 102 includes an image sensor, a lens, a filter, and the like, and captures an image including a user operation unit (hand or finger). The illuminations 103 and 104 include a light emitting diode, a circuit board, a lens, and the like, and irradiate an area captured by the camera 102. The lights 103 and 104 may be constantly lit, but the lights 103 and 104 may be lit alternately. Further, when the lighting 103 and 104 is turned on, both may be temporarily turned off, or the lighting 103 and 104 may be blinked at the same timing. The illumination light may be invisible light. For example, the camera 102 and the illuminations 103 and 104 may be configured by an infrared camera and infrared illumination, and an operation detection process may be performed from the captured infrared image. In that case, it is preferable to add a filter to the infrared camera and shoot with some or all of the light not in the infrared region blocked. That is, by separating the wavelength region of the illumination light from the illuminations 103 and 104 from the wavelength region of the light projected by the display function unit 151 (image projection unit 153), the user can project the projected image without visually recognizing the illumination light. Can be watched.

The operation detection unit 105 includes a circuit board, software, and the like, and detects the operation content from the image of the operation article photographed by the camera 102. The operation object can be a user's finger or hand or an operation pen. In addition, an image of a user's characteristic hand is extracted from an image captured by the camera 102 and used for user identification. In this embodiment, an image of the user's hand is used as the user's feature.

The user management unit 106 has a circuit board, software, and the like, and registers the features of each user's hand in the database as user data. Further, each user is managed by giving an attribute indicating a user ID and a user operation authority.

The user identification unit 107 includes a circuit board, software, and the like. The user ID of the user who performs the operation by collating the hand image captured by the camera 102 with the user data registered in the database of the user management unit 107 And attributes.
The operation detection unit 105, the user management unit 106, and the user identification unit 107 described above are portions that detect a user operation and identify a user who operates, and are referred to as a user-specific operation detection unit 110.

The communication unit 108 includes a network connection, a USB connection, an ultrasonic unit, an infrared communication device, and the like, and is an interface capable of communicating with a device outside the detection function unit 101 such as a display or a projector.

The control unit 109 includes a circuit board, software, and the like, and controls the camera 102, the illuminations 103 and 104, the operation detection unit 105, the user management unit 106, the user identification unit 107, and the communication unit 108.

The detection result data 121 is data that the detection function unit 101 outputs to the display function unit 121 via the communication unit 108, and includes information such as the detected operation content, the user ID of the operated user, and the user attribute.

Next, each component of the display function unit 151 will be described. The display function unit 151 receives the detection result data 121 and displays an image according to an operation with a user's finger or pen.

The communication unit 152 includes a network connection, a USB connection, an ultrasonic unit, an infrared communication device, and the like, and is an interface that can communicate with components outside the display function unit 151 such as a display and a projector.

The image projection unit 153 includes a light source lamp, a liquid crystal panel, a lens, and the like, and projects and displays an image on a projection surface. Here, if the wavelength region of the light projected from the image projection unit 153 does not overlap the wavelength of the illumination light (infrared light) from the illuminations 103 and 104 of the detection function unit 101, the camera (infrared camera) The user's operation image captured at 102 is not affected. For this purpose, a band pass filter or the like for controlling the wavelength of the projection light may be used.

The control unit (image processing unit) 154 includes a circuit board, software, and the like, and controls the communication unit 152 and the image projection unit 153. In particular, the image projected from the image projection unit 153 is processed according to the detection result data 121 sent from the detection function unit 101. That is, whether to accept an operation, the type of video to be displayed, the display format, and the like are determined according to the user ID and user attribute of the operated user as well as the user operation content.

The video display device shown in FIG. 1 includes the detection function unit 101 and the display function unit 151. However, the detection function unit 101 may be made independent to be an operation detection device. The elements 102 to 109 and 152 to 154 are independent, but may be configured with one or a plurality of constituent elements as necessary. For example, the elements 105 to 107 may be configured to perform their processing by one or more central processing units (CPUs). The elements 102 to 109 are all included in the detection function unit 101, and the elements 152 to 154 are all included in the display function unit 151. You may couple | bond by a universal serial bus (USB) connection. For example, the elements 102 to 104 may be configured outside the detection function unit 101.

FIG. 2 is a diagram showing the appearance of the projection display apparatus and the operation state of the user. (A) is a front view of the operation state, (b) is a side view of the operation state. The user 251 operates the image projected on the projection plane by the image projection unit 153 using the finger 252. The operation surface 201 overlaps at least a part of the image projection surface projected by the image projection unit 153, and the user brings a part of the projected image close to the finger, for example, and touches a certain position. To give a desired operation instruction. The camera 102 images the area of the operation surface 201 and irradiates illumination light from the illuminations 103 and 104 for imaging. In accordance with a user operation instruction, the image projected by the image projection unit 153 is controlled, and the display format of the image is changed or switched to another image.

In FIG. 2, the user operates with a finger, but the user may perform an operation using an operation tool such as an operation pen or a bar instead of the finger. Or you may operate by not only a finger but the whole hand. Further, FIG. 2 shows a case where there is one user, but it is also possible for a plurality of users to perform operations simultaneously or by switching.

FIG. 3 is a diagram showing a difference in shadow shape depending on whether or not a finger and an operation surface are in contact. (A) and (b) are a front view and a top view when the finger 400 is not in contact with the operation surface 201, and (c) and (d) are views when the fingertip of the finger 400 is in contact with the operation surface 201. A front view and a top view in the case of being present are shown.

In the case of non-contact as shown in (a), a shadow 401 projected by the illumination 104 and a shadow 402 projected by the illumination 103 are formed on both sides of the finger 400, respectively. It will be in a separated state. (B) explains the principle of creating a shadow as shown in (a). When viewed from the direction of the tip of the finger 400, the light emitted by the illumination 104 is blocked by the finger 400, and a shadow 401 is formed on the operation surface 201. Further, the light emitted by the illumination 103 is blocked by the finger 400, and a shadow 402 is formed on the operation surface 201. Since the finger 400 is away from the operation surface 201, the shadow 401 and the shadow 402 are separated from each other in the image captured by the camera 101.

Next, when they are in contact as shown in (c), the shadow 401 and the shadow 402 are close to each other. (D) explains the principle of creating a shadow as shown in (c). When viewed from the direction of the tip of the finger 400, a shadow 401 is formed by the light emitted by the illumination 104, and a shadow 402 is formed by the light emitted by the illumination 103. Since the finger 400 is in contact with the operation surface 201, the shadow 401 and the shadow 402 are close to each other in the image captured by the camera 102.

FIG. 4 is a diagram illustrating the relationship between the distance between the finger and the operation surface and the shape of the shadow.
(A) shows how the shape of the shadow changes depending on the distance between the finger 400 and the operation surface 201. When the distance between the finger 400 and the operation surface 201 is the shortest (at the time of contact), the shadow 401 and the shadow 402 are in the closest state. When the finger 400 and the operation surface 201 are separated, the distance between the shadow 401 and the shadow 402 gradually increases, and the distance between the shadow 401 and the shadow 402 depends on the distance between the finger 400 and the operation surface 201. Therefore, by measuring the distance between the shadow 401 and the shadow 402, the distance between the finger 400 and the operation surface 201 and the contact / non-contact can be determined.

(B) to (e) indicate at which position of the shadow (called a feature point) the distance between the shadow 401 and the shadow 402 is measured. In (b), feature points 403 and 404 are set at the tip (fingertip) of the shadow, and the distance between the shadow 401 and the shadow 402 is defined by the distance d between the feature points 403 and 404. In (c), feature points 403 and 404 are set outside the portion corresponding to the shadow finger, and the distance between the shadow 401 and the shadow 402 is defined by the distance d between the feature points. Alternatively, as shown in (d), the feature points 403 and 404 are set on the outermost part of the shadow, or on the outer side of the part corresponding to the shadow wrist as shown in (e). The method of setting is also acceptable.

Next, processing of the operation detection unit 105, the user management unit 106, and the user identification unit 107 in the user-specific operation detection unit 110 will be described in detail.

FIG. 5 is a diagram illustrating processing for detecting a contact point between the finger and the operation surface in the operation detection unit 105. (A) shows a processing flow, (b) is a figure explaining the process of S507 and S508.

In step S <b> 501, it is determined whether or not two shadows related to the finger 400 are detected in the image captured by the camera 102. If detected, the process proceeds to S502.
In S502, a feature point 403 and a feature point 404 are detected in the two shadows 401 and 402 related to the finger (see FIG. 4).
In S503, the distance d between the feature point 403 and the feature point 404 is measured.

In S504, it is determined whether the distance d between the feature point 403 and the feature point 404 is smaller than a predetermined value. If it is smaller than the predetermined value, the process proceeds to S505, and if it is greater than the predetermined value, the process proceeds to S506.
In S505, it is determined that the finger 400 is in contact with the operation surface 201, and the process proceeds to S507.
In S506, it is determined that the finger 400 is not in contact with the operation surface 201, and the flow ends.

In S507, as shown in FIG. 5B, the tip 541 and the tip 542 are detected in the two shadows 401 and 402 relating to the finger, respectively.
In S508, as shown in FIG. 5B, the midpoint 551 of the tip 541 and the tip 542 is detected as a contact point between the finger 400 and the operation surface 201. Then, the position (coordinates) of the contact point 551 is read, and the process ends.

Even if the above flow is completed, if the user is continuing the operation, the flow returns to S501 to repeat the above flow in order to detect the next contact point.
The processing in the operation detection unit 105 may use another image processing algorithm for obtaining a similar result.

FIG. 6 is a diagram showing user registration processing in the user management unit 106. The user registration process is a process of registering information (user data) related to a user who operates the operation surface 201 in a database, (a) shows a processing flow, and (b) shows contents of user data to be registered. . The user registration process shown here is a case in which the user management unit 106 automatically starts with the detection of a user who is not registered in the database as a trigger.

In S601, the camera 102 captures an image of each user's hand and extracts features relating to the hand. Features relating to the hand include the size, shape, color, and the like of the hand, and details will be described with reference to FIG. As shown in FIG. 2, when the user 251 operates the operation surface 201, the user's hand always exists within the shooting range of the camera 102, and the feature of the user's hand is extracted from the captured image. be able to. That is, the captured image of the camera 102 can be used for both user operation detection and acquisition of user hand characteristics.

In S602, the extracted features of the user's hand are registered in the database. As shown in FIG. 6B, features 600a to 600c regarding the hands of the users A to C are registered as user data. Here, an image (outline) of each user's hand is described, but the features of the hand may be digitized or documented as described later. The registered user feature is used in a user identification process described later.

In S603, identification information (user ID) is given to the user. The user ID assigned here is a user name or a user number for individually identifying the user when there are a plurality of users, and preferably does not overlap. The assigned user ID is registered in the database in association with the user feature registered in S602. In the case of automatically assigning a user ID, for example, “user A”, “user B”,...

In S604, an attribute is given to the user. The attribute is information for classifying the users according to common characteristics, and here, the users are classified according to the operation authority. For example, an attribute of “special” is given to a user who is permitted all operations, and a “general” attribute is given to a user who is restricted to some operations. In the following, a user with the “special” attribute is called a “special user”, and a user with the “general” attribute is called a “general user”. The assigned attribute is registered in the database in association with the user feature registered in S602 or the user ID registered in S603. If it is not necessary to classify users by attribute, S604 may be omitted.

The attribute can be automatically set according to the usage status of the user. For example, a user who first receives an attribute is a special user, and a user who receives an attribute after the second is a general user. Alternatively, a user whose usage frequency is a certain value or more is a special user, and other users are general users. Alternatively, in consideration of the use in school or the like, a user (teacher) whose hand size is a certain value or more may be a special user, and other users (students) may be general users. Alternatively, after all users are uniformly set as general users, some users may be manually changed to special users.
Through the above processing, a database having user data as shown in FIG. 6B is created. The user data is composed of user characteristics, user IDs and attributes of each user.

FIG. 7 is a diagram showing examples of parameters representing the features of the user's hand. (A) is the size of the hand, (b) is the length of the finger, and (c) is the thickness of the finger. (D) is a polygonal approximation of the hand shape, and a plurality of hand feature points are defined and connected. (E) is a polygonal approximation of the shape of the finger, which defines a plurality of finger feature points and connects them. (F) is a blood vessel pattern. Since blood has a high absorption rate of infrared light, a clear blood vessel pattern can be obtained from an image taken by the camera 102, particularly when the infrared light is irradiated from the illuminations 103 and 104. Other parameters such as hand color, hand wrinkle pattern, hand luminance distribution, and nail shape can also be used as hand features.

The format of user data to be registered in the database may be hand image data, but the above parameters are digitized or documented for ease of use. For example, for the hand shape of (d), the coordinates of the vertexes of the polygon may be registered. Moreover, one of the above parameters may be registered in the database, or a plurality of parameters may be combined from these parameters.

FIG. 8 is a diagram showing a manual user registration processing flow. In this case, the user registration process is started by the user management unit 106 triggered by the user's activation of the user registration mode. In addition, S802 to S809 in the following processing flow show examples of screens displayed on the operation surface 201. Even when the user manually performs the registration process, the operation detection unit 105 detects the contact point of the user's finger and determines the operation content.

In S801, the user activates the user registration mode.
In step S802, the user registration is notified.

In S803, the target is displayed and the user's right hand is guided to a predetermined position. When the user moves the right hand to a predetermined position, the camera 102 captures the situation. The user management unit 106 extracts features of the user's right hand from the captured image.
In S804, the extracted feature of the right hand of the user is registered in the database (corresponding to S602 in FIG. 6), and notification that the registration of the right hand has been completed is sent.

In S805, the target is displayed and the user's left hand is guided to a predetermined position. When the user moves the left hand to a predetermined position, the camera 102 captures the situation. The user management unit 106 extracts the left hand feature of the user from the captured image.
In S806, the extracted feature of the left hand of the user is registered in the database (corresponding to S602 in FIG. 6), and notification that the registration of the left hand has been completed.

In S807, the user is requested to input a user name. The user name input here is registered as a user ID in the database (corresponding to S603 in FIG. 6).
In S808, the user is requested to select a classification of the user. The classification of the user selected here is registered in the database as the attribute of the user (corresponding to S604 in FIG. 6). If there is no need to give an attribute to the user, S808 may be omitted.
In step S809, the user registration is notified. When the notification is completed, the user registration process flow is terminated.

In addition, even if the above-described flow is completed, if another user registration is to be continued, the process returns to S801 and the above-described flow is repeated. In addition, when a user who has already been registered in the database tries to register again, a route through the flow may be provided in order to avoid duplication of user data. In addition, when registering the feature of only one user's hand, one of S803 to S804 and S805 to S806 may be executed.

In the automatic user registration process described in FIG. 6, no user operation is required for the registration process, and there is no burden on the user. On the other hand, in the manual user registration process described with reference to FIG. 8, the user's hand is guided to a predetermined position and imaged, and the user himself inputs the user name and selects the user type. User data can be acquired. Whether the user registration process is performed manually or automatically may be appropriately selected according to the environment in which the video display device is used and the user's preference.

FIG. 9 is a diagram showing a user identification processing flow in the user identification unit 109.
In step S <b> 901, it is determined whether a user who operates the operation surface 201 is detected in an image captured by the camera 102. If a user is detected, the process proceeds to S902.

In step S902, features of the user's hand are extracted based on the user image detected in step S901. And it collates with the database of the user management part 106. FIG. At the time of collation, the extracted user characteristics are compared with user data registered in the database.
In step S903, it is determined whether there is user data (user feature) that matches the detected user feature in the database as a result of collation. If there is matching user data, the process proceeds to S904, and if not, the process proceeds to S906.

In S904, the user ID of the matched user data is referred to, and the user ID of the detected user is identified. For example, if the detected feature of the user matches the user feature 600a in FIG. 6B, the user ID is “user A”.
In step S905, the attribute of the detected user is referred to by referring to the attribute of the matched user data. For example, if the detected feature of the user matches the user feature 600a of FIG. 6B, it can be seen that the attribute of the user is “special”. If no attribute is given to the user (such as when S604 is not performed), S905 may be omitted. When the identification is completed, the identification processing flow ends.

In S906, it is determined that the detected user is an unregistered user.
In S907, it is determined whether or not to perform user registration for the detected user. In this case, the user may be instructed. If user registration is to be performed, the process advances to step S908; otherwise, the identification processing flow is terminated.

In S908, the detected user is registered by the above-described user registration process (S601 to S604 in FIG. 6). Alternatively, manual user registration processing (S802 to 809 in FIG. 8) may be performed.

By the above-described user identification process, it is possible to know whether or not the operated user is registered and, if registered, the user ID and attribute. If the user ID and attribute are known, video display is controlled according to a rule separately defined for each user ID and attribute.

Next, some examples of identifying an operating user and controlling video display according to the user ID and the user attribute will be described.

FIG. 10 is a diagram illustrating an example in which different images are projected to different users. Here, three users are identified, and the image projection unit 151 projects different images (image patterns) 1000a and 1000b so as to be superimposed on the hands of the users A and B. The projected video pattern has a unique shape and color that is predetermined for each user ID or attribute. A character string may be projected instead of the image. Further, the image may not be projected to a specific user (for example, an unregistered user).

Thus, by projecting a predetermined video according to the user ID and attribute, each user confirms whether or not his / her user data (user characteristics, user ID and attribute) is correctly registered in the database. be able to.

FIG. 11 is a diagram showing an example in which video is displayed in different display formats for different users. Here, the drawing trajectories (operation histories) 1100a to 1100c for the drawing operations performed by the users A to C are displayed. The trajectory to be displayed has different display formats such as line thickness, color, and type for each user ID and attribute. The display format assigned to each user A to C is automatically set on the video display device side.

In this way, by displaying the drawing trajectory in different formats according to the user ID and attributes, it is possible to easily identify which user is the user even if drawing by a plurality of users is mixed.

FIG. 12 is a diagram showing an example of displaying different menus for different users. The menu displayed here is a tool for assisting the user's operation, and corresponds to a display format selection branch or the like. On the operation surface 201, different menus 1200a to 1200c are displayed for the users A to C who are performing the operation. These menus 1200 include, for example, an interface for changing the thickness, color, and type of a line for displaying a drawing trajectory, an interface for deleting a line once displayed, and the like. The menus 1200a to 1200c to be displayed have different items and arrangement for each user ID or for each attribute. Therefore, when the type of operation permitted for each user is different, only the operation items permitted for each user are displayed on the menu. Or you may enable it to set the item and arrangement | positioning of a menu according to a user preference. The display position of the menu in the operation surface 201 is preferably displayed close to the user so as to follow the movement of the user.

By displaying different menus according to user IDs and attributes in this way, it is possible not only to identify which user is drawing by a plurality of users, but also it is easy for the user to change the display format during drawing. .

FIG. 13 is a diagram illustrating an example in which different display processes are performed for different users. Here, the process of deleting what the user once described is taken up.

(A) is a state in which user A and user B have described the characters 1300a and 1300b, respectively. (B) shows the processing when user A performs the delete 1301 operation after the description in (a), only the description 1300a by user A is deleted, and the description 1300b by user B remains without being deleted. ing. On the other hand, (c) shows processing when user B performs an erase 1301 operation after the description of (a), only the description 1300b by user B is deleted, and the description 1300a by user A is not deleted. Remaining. That is, the operation (deletion process) performed by each user is applied only to the operation history (description) performed by the user.

By managing the operation history of each user in association with the user ID, the processing as shown in FIG. 13 can be performed. As a result, even if a plurality of users operate at the same time, the operations of each user do not compete, and the operation history performed by other users is not affected.

FIG. 14 is a diagram illustrating an example of restricting operations according to user attributes. Here, an example of adding a restriction to the highly confidential processing X1400 will be described.

(A) A special user (attribute = special) is performing an operation for executing the process X1400. As a result, the process X is normally accepted, and a message 1401 to that effect is displayed in (b).

In (c), a general user (attribute = general) performs an operation for executing the process X1400. However, processing X is restricted to special users only, and processing X is not accepted. At this time, in order to notify the general user that the process X cannot be executed, a message 1402 such as “General user cannot execute the process X” is displayed as shown in FIG.

¡It is set in advance what kind of operation is allowed for each attribute. As a result, for example, by limiting the processing with high confidentiality and difficulty to special users, accidents such as leakage of confidential information can be prevented.

FIG. 15 is a diagram illustrating another example in which operations are restricted according to user attributes. In this example, although a plurality of users perform drawing operations at the same time, only a special user drawing operation is accepted and a general user drawing operation is not accepted. At this time, in order to notify the general user that the process cannot be executed, for example, a message 1500 such as “Only a special user operation is accepted” may be displayed.

Thus, giving the operation authority according to the user attributes (or setting the operation priority order) can prevent the processing from being confused by a plurality of operations when a large number of users perform the operation simultaneously. For example, when a teacher and a large number of students operate at the same time in a school or the like, this is effective when priority is given to the teacher's operation.

10 to 15, even when an unregistered user who has not been registered is performing an operation, a user ID or attribute is provisionally assigned based on the user's characteristics and registered. You may treat it like any other user.

As described above, in the first embodiment, the camera 102 captures the user's hand, and the operation detection unit 105 detects the user's operation. On the other hand, the user management unit 106 manages hand features, user IDs, and attributes for each user, and the user identification unit 107 identifies user IDs and attributes from the features of the user's hand.

As a result, it becomes possible to switch the processing contents according to the user ID and attribute, and even when a plurality of users operate at the same time or by switching, the projection-type video display device is stable and easy to use without confusion. Can provide.

In Example 2, a user's hand and face are photographed with a camera for user identification. Then, each user is identified based on the hand and face characteristics, and the video display is controlled.

In Example 2, the configuration of the projection display apparatus is the same as that of Example 1 (FIG. 1). In the first embodiment, the user's hand is photographed by the camera 102, whereas in the second embodiment, the user's hand and face are photographed. When it is difficult to photograph both hands and faces with one camera, a plurality of cameras may be prepared, and one of them may photograph a hand and the other with one. Alternatively, the number of cameras may be one, and a wide-angle lens or a fisheye lens may be attached to the camera to expand the shooting range so that both hands and faces can be shot.

In the user-specific operation detection unit 110, the operation detection unit 105 detects the user's operation and extracts an image of the user's hand and face from the image taken by the camera 102 for user identification. The user management unit 106 registers the hand and face characteristics of each user in the database as user data. The user identification unit 107 compares the hand and face images captured by the camera 102 with user data registered in the database, and identifies the user who is performing the operation.

Processing for detecting a contact point between the finger and the operation surface in the operation detection unit 105 is the same as that in the first embodiment (FIG. 5). The user registration process in the user management unit 106 is performed in the same manner as in the first embodiment (FIG. 6). However, in S601 and S602, the features of the user's hand and face are taken up as user features.

FIG. 16 is a diagram illustrating an example of user data managed by the user management unit 106. In the second embodiment, features 600a to 600c relating to the user's hand and features 1600a to 1600c relating to the face of the user are extracted based on the images of the hands and faces of the users A to C photographed by the camera 102 and registered in the database. The parameters representing the features of the user's hand are as described in the first embodiment (FIG. 7). The parameters representing the facial features 1600a to 1600c to be added include the relative positions and sizes of facial parts, facial contours, facial colors, eyes, nose, mouth, and chin shapes.

The features related to the user's hand and face registered in the database are used in the user identification process. The user registration process can be performed either manually or automatically, but may be selected according to the environment in which the video display device is used or the user's preference.

As can be seen from the operation mode of the video display device shown in FIG. 2, when the user 251 operates the operation surface 201, the user's hand always exists within the shooting range of the camera 102. However, the user's face is not necessarily within the shooting range of the camera 102 depending on the user's standing position and the direction in which the camera is directed. Therefore, when registering the facial features of the user, the registration process must be performed after waiting for the user's face to fall within the shooting range. In such a case, the user's face should be registered manually.

FIG. 17 is a diagram showing a manual user registration processing flow. The manual user registration process is triggered by the user's activation of the user registration mode.

In the manual user registration process, the face registration process of S1701-1703 is added to the hand registration process of S801-809 described in the first embodiment (FIG. 8). The position to be added is, for example, between S802 and S803. The user's hand registration process may be performed with only one hand. S1701 to S1703 represent screens displayed on the operation surface 201, and the user performs an operation for user registration processing according to the projection screen.

In step S1701, the user is notified to view the camera.
In S1702, a camera image 1711 being captured by the camera 102 is displayed, and the user's face is guided to a predetermined position. When the user moves the face to a predetermined position, the camera 102 captures the user's face.
In step S1703, notification that the face registration is completed is sent.

Thereafter, the user's hand characteristics are registered in S803 to S806, and in S807 and S808, the user ID and attributes are assigned to the user's hand and face characteristics to complete the user registration.

User identification processing in the user identification unit 107 is the same as that in the first embodiment (FIG. 9). However, in S902 (user feature verification), the detected hand feature and facial feature are combined, and the hand and facial features registered in the database are compared. The use of both hand and face features for matching reduces the probability of matching errors and has the effect of further improving the reliability of the identification process.

In the operation mode of the video display apparatus shown in FIG. 2, the user's face being operated does not necessarily exist within the shooting range of the camera 102. If the user's face cannot be photographed, collation may be performed using only the hand feature, and if the user's face can be photographed, collation may be performed using both the hand feature and the facial feature.

Needless to say, also in the second embodiment, various processes based on user identification shown in FIGS. 10 to 15 of the first embodiment are similarly realized.

Also in the second embodiment, it is possible to provide a stable and easy-to-use video display device without confusion in control even when a plurality of users are used simultaneously or after replacement. In particular, in the second embodiment, since the user's hand feature and the facial feature are used in combination, the user identification accuracy is improved and the reliability of the apparatus can be further improved.

In Example 3, a user's hand and a wearable terminal worn by the user are photographed with a camera for user identification. And each user is identified based on the characteristic of a hand and a wearable terminal, and video display is controlled.

Here, the wearable terminal is a terminal that can be worn on the body such as a user's finger, wrist, head, etc. For example, a general wearing tool such as a ring, a wristwatch, and glasses, or only an outer shape has the shape of the wearing tool. It may be a thing. Further, the wearable terminal may have a communication function. Also, the wearable terminal worn by the user may be owned by each user or may be assigned to each user for the operation of the video display device.

In the third embodiment, the configuration of the projection display apparatus is the same as that of the first embodiment (FIG. 1). In the third embodiment, the user's hand and wearable terminal are photographed by the camera 102. If it is difficult to shoot both hands and wearable terminals with one camera, prepare multiple cameras, and use one of them to shoot your hand and the other to shoot your wearable terminal. Good. Alternatively, the number of cameras may be one, and a wide-angle lens or a fisheye lens may be attached to the camera to expand the shooting range so that both hands and wearable terminals can be shot.

In the user-specific operation detection unit 110, the operation detection unit 105 detects the user's operation and extracts the user's hand and the wearable terminal image from the image taken by the camera 102 for user identification. The user management unit 106 registers the hand of each user and the characteristics of the wearable terminal in the database as user data. The user identifying unit 107 collates the hand and wearable terminal image captured by the camera 102 with user data registered in the database, and identifies the user who is performing the operation.

Processing for detecting a contact point between the finger and the operation surface in the operation detection unit 105 is the same as that in the first embodiment (FIG. 5). The user registration process in the user management unit 106 is performed in the same manner as in the first embodiment (FIG. 6). In S601 and S602, the user's hand and the wearable terminal features are registered as user features.

FIG. 18 is a diagram illustrating an example of user data managed by the user management unit 106. In the third embodiment, features 600a to 600c relating to the user's hand and features 1800a to 1800c relating to the wearable terminal are extracted based on the images of the hands of the users A to C and the wearable terminals (here, wristwatches) captured by the camera 102. And register it in the database. The parameters representing the features of the user's hand are as described in the first embodiment (FIG. 7). The parameters representing the wearable terminal features 1800a to 1800c to be added include the size, shape, color, etc. of the terminal (watch).

The features related to the user's hand and face registered in the database are used in the user identification process. The user registration process can be performed either manually or automatically, but may be selected according to the environment in which the video display device is used or the user's preference. Hereinafter, manual user registration processing will be described.

The manual user registration process is the same as that of the first embodiment (FIG. 8) when the wearable terminal is worn on the hand. It is assumed that each user wears a wearable terminal on his / her right hand or left hand. In step S803 if the user wears the right hand, and in step S805 if the user wears the left hand, the user's hand and the wearable terminal are also photographed to extract the hand features and wearable terminal features as user features. Register in the database.

Note that when the wearable terminal is worn on the face (for example, glasses), the processing of the second embodiment (FIG. 17) may be applied. That is, in step S1702, the user's face may be photographed, and the wearable terminal (glasses) portion may be extracted from the image.

User identification processing in the user identification unit 107 is the same as that in the first embodiment (FIG. 9). However, in S902 (user feature collation), the detected features of the user's hand and the features of the wearable terminal are combined, and the features registered in the database and the features of the wearable terminal are compared. By using the characteristics of both the hand and the wearable terminal at the time of collation, the probability of collation error is reduced, and the reliability of the identification process is further improved.

Note that if the camera cannot be used to shoot the wearable terminal, such as when the user is operating with a hand that does not wear the wearable terminal, the collation is performed using only the hand features, and if the wearable terminal can be shot, You may make it collate using both the characteristic and the characteristic of a wearable terminal.

In this embodiment, since the collation is performed using a wearable terminal having a feature in shape and color, the user identification function is improved as compared with the case of using a hand or face.

Furthermore, when the wearable terminal has a communication function, the user can be identified using the communication function. In the user registration process in that case, the detection function unit 101 communicates with the wearable terminal in advance via the communication unit 108 and acquires terminal ID information that can identify the terminal from each wearable terminal. The user management unit 106 registers the acquired terminal ID information in the database together with the user ID (see FIG. 18). The user identification unit 107 is registered in the database based on not only the characteristics of the wearable terminal worn by the user extracted from the camera image, but also the terminal ID information acquired by the detection function unit 101 communicating with the wearable terminal. The user is identified by collating with user data (characteristics of wearable terminal and terminal ID information). By using the terminal ID information, the accuracy of user identification is further improved.

Needless to say, also in the third embodiment, various processes based on user identification shown in FIGS. 10 to 15 of the first embodiment are similarly realized.

Also in the third embodiment, it is possible to provide a stable and easy-to-use video display device without confusion of control even when a plurality of users are used at the same time or in exchange. Particularly in the third embodiment, since the features of the user's hand and the features of the wearable terminal to be worn are used in combination, the identification accuracy of the user is improved and the reliability of the apparatus can be further increased.

Example 4 describes user identification when a user uses an operation pen for operation. In addition to capturing the user's hand by photographing the user's hand with a camera, each user is identified based on whether or not the hand is holding a pen and video display is controlled. Here, the structure of the pen for operation is not particularly limited, but may be one that emits light upon contact as described later. Moreover, it is not necessary to distinguish the pen to be used for each user.

In Example 4, the configuration of the projection display apparatus is the same as that of Example 1 (FIG. 1). In the fourth embodiment, the user's hand is photographed by the camera 102, and when the user's hand is holding the pen, the pen is also photographed.

In the user-specific operation detection unit 110, the operation detection unit 105 detects the user's operation and extracts the image of the user's hand and pen from the image taken by the camera 102 for user identification. The user management unit 106 registers the features of each user's hand and the presence or absence of a pen in the database as user data. The user identification unit 107 compares the hand and pen images taken by the camera 102 with user data registered in the database, and identifies the user who is performing the operation.

The operation detection unit 105 performs processing for detecting a contact point between the pen and the operation surface. In addition, when the user who does not hold | maintain a pen contacts an operation surface with a finger, it is the same as that of Example 1 (FIG. 5).

FIG. 19 is a diagram illustrating processing for detecting a contact point between the pen and the operation surface. The pens to be used will be described separately for a type in which the tip does not emit light when contacting the surface (non-light emitting pen) and a type that emits light when contacting the surface (light emitting pen or electronic pen).

(A) shows a state in which the camera 1102 shoots a pen 1951 whose tip does not emit light when contacting the surface. As in the case of the shadow of the finger 400 described with reference to FIGS. 3 to 4, when the user holds the pen 1951, a shadow 1901 due to the illumination 104 and a shadow 1902 due to the illumination 103 are generated on both sides of the pen 1951. The distance between the two shadows 1901 and 1902 decreases as the pen 1951 approaches the operation surface 201. Here, feature points 1903 and 1904 are set at the tips of the respective shadows, and the distances of the shadows 1901 and 1902 are defined by the distance d between the feature points. When the distance d between the feature points is smaller than a predetermined value, it is determined that the tip of the pen 1951 is in contact with the operation surface 201. At this time, a midpoint 1905 of the feature points 1903 and 1904 is detected as a contact point between the pen 1951 and the operation surface 201.

(B) shows a state in which the light emitting pen 1952 whose tip emits light when contacting the surface is photographed by the camera 102. The light-emitting pen 1952 includes a pressure-sensitive sensor, a light-emitting element, a circuit board, and the like, and detects whether or not the tip is in contact with the surface by the pressure-sensitive sensor. When the tip of the light emitting pen 1952 is in contact, the light emitting element is turned on, and when it is not in contact, the light emitting element is turned off. When the light emitting pen 1952 emits light by a user operation, a light emitting area 1906 is formed at the tip. At this time, the center position of the light emitting area 1906 is detected as a contact point 1907 between the light emitting pen 1952 and the operation surface 201.
The processing in the operation detection unit 105 may use another image processing algorithm for obtaining a similar result.

The user registration processing in the user management unit 106 is performed in the same manner as in the first embodiment (FIG. 6) except for S604. However, in the attribute assignment in S604, a case where setting is performed based on the presence or absence of a pen will be described.

FIG. 20 is a diagram illustrating an example of user data managed by the user management unit 106. In the first embodiment, attributes such as “special” or “general” are fixedly given to each user. On the other hand, in Example 4, the attribute for each user is not fixed, but the attribute is determined according to the situation. As an attribute determination rule for that purpose, here, when the user is holding the pen, it is treated as “special user”, and when the user is not holding the pen, it is treated as “general user”.

The user registration process can be either manual or automatic, but any one may be selected according to the environment in which the video display device is used and the user's preference. In the manual user registration process, the attribute determination process (S808) in the first embodiment (FIG. 8) is not selected by the user, but is automatically determined based on the presence or absence of a pen from the user image.

User identification processing in the user identification unit 107 is the same as that in the first embodiment (FIG. 9). However, in S905 (attribute identification), the attribute is identified in accordance with the user attribute determination rule registered in the user data of FIG. That is, the user is identified as a special user when the pen is held, and as a general user when the pen is not held.

In this embodiment, the user is identified from the characteristics of the user's hand, and the attribute is identified based on whether or not the user is holding the pen. Since the user holding the pen is treated as a special user, the operation priority is higher than that of a general user not holding the pen. Therefore, the special user who gives priority to the operation can be easily changed by changing the pen user among a plurality of users.

Needless to say, in the fourth embodiment, various processes based on user identification shown in FIGS. 10 to 15 of the first embodiment are similarly realized.

Also in the fourth embodiment, it is possible to provide a stable and easy-to-use video display device without confusion in control even when a plurality of users are used at the same time or in exchange. In particular, in the fourth embodiment, when the user operates using the pen, the setting of the user prioritizing the operation can be easily changed, and the usability is further improved.

101: Function detection unit (operation detection device),
102: Camera (imaging unit),
103, 104: lighting,
105: Operation detection unit,
106: User management unit,
107: a user identification unit,
108: Communication unit
109: Control unit,
110: User-specific operation detection unit,
121: detection result data,
151: Display function unit,
152: Communication unit,
153: image projection unit,
154: Control unit (video processing unit),
201: operation surface,
251: user,
252,400: finger,
401, 402: shadow,
403, 404: feature points,
551: contact point,
600, 1600: user characteristics,
1800: Wearable terminal,
1951, 1952: pens for operation.

Claims (13)

In a projection display apparatus that projects and displays an image on a projection surface,
An imaging unit for imaging the image projection area of the projection surface;
An operation detection unit that detects a user's operation content for video display based on an image captured by the imaging unit;
A user identification unit for identifying a user who has performed the operation based on an image captured by the imaging unit;
An image processing unit for processing an image displayed on the projection surface in accordance with the user operation content and the identification result of the user;
An image projection unit that projects the image processed by the image processing unit onto the projection plane;
A projection-type image display device comprising: The projection display apparatus according to claim 1,
For each user who performs the operation, a user management unit that registers, as user data, characteristics of at least one image of the user's hand, face, and attachment in advance,
The user identification unit compares a user image captured by the imaging unit with user data registered in the user management unit to identify a user who has performed the operation. Display device. A projection display apparatus according to claim 2,
In the user management unit, the user who performs the operation is classified according to operation authority, an attribute corresponding to the operation authority is assigned to each user and registered in the user data,
The projection processing apparatus, wherein the image processing unit performs processing of an image displayed on the projection plane in accordance with a user who performs the operation and an attribute given to the user. In an image display method for projecting and displaying an image on a projection surface,
An imaging step of imaging the image projection area of the projection surface;
An operation detection step for detecting a user's operation content for video display based on the image captured by the imaging step;
A user identification step for identifying the user who performed the operation based on the image captured by the imaging step;
An image processing step for processing an image to be displayed on the projection surface in accordance with the user operation content and the identification result of the user;
A video display method comprising: The video display method according to claim 4,
For each user who performs the operation, a user registration step of registering, as user data, characteristics of at least one image of the user's hand, face, and attachment in advance,
In the user identification step, the image of the user imaged in the imaging step and the user data registered in the user registration step are collated to identify the user who performed the operation, . The video display method according to claim 4 or 5,
Categorizing users performing the operation according to operation authority, and providing an attribute granting step for assigning attributes according to the operation authority;
In the image processing step, an image displayed on the projection plane is processed according to a user who performs the operation and an attribute given to the user. The video display method according to claim 4 or 5,
A video display method characterized in that, in the video processing step, a process for displaying a predetermined video pattern specific to the user is performed using the operation object of the user who performed the operation as a projection plane. The video display method according to claim 5,
If the user is wearing a terminal that can communicate,
In the user registration step, the terminal ID of the terminal is registered in advance as user data of the user,
In the user identification step, communication with a terminal of the user who performs the operation is performed to obtain a terminal ID of the terminal, and the user who performed the operation is identified by collating with a terminal ID of the registered user data. Characteristic video display method. The video display method according to claim 6,
In the operation detection step, when it is detected that the user who performs the operation holds the operation pen,
In the attribute assigning step, a video display method is characterized in that an attribute of an operation authority higher than that of a user who does not hold the operation pen is given to a user who holds the operation pen. In an operation detection device for detecting a user operation on a projection plane of an image,
An imaging unit for imaging the image projection area of the projection surface;
An operation detection unit that detects a user's operation content based on an image captured by the imaging unit;
A user identification unit for identifying a user who has performed the operation based on an image captured by the imaging unit;
An output unit that outputs the identification result of the user who performed the operation together with the detected operation content of the user,
An operation detection apparatus comprising: The operation detection device according to claim 10,
For each user who performs the operation, a user management unit that registers, as user data, characteristics of at least one image of the user's hand, face, and attachment in advance,
The operation identifying device characterized in that the user identifying unit collates a user image captured by the imaging unit and user data registered in the user managing unit to identify a user who performed the operation. . The operation detection device according to claim 11,
In the user management unit, the user who performs the operation is classified according to operation authority, an attribute corresponding to the operation authority is assigned to each user and registered in the user data,
An operation detection apparatus that refers to the user data and outputs attribute information of the user together with an identification result of the user who performed the operation from the output unit. The operation detection device according to claim 12,
When the operation detecting unit detects that the user performing the operation is holding the operation pen,
The operation management apparatus according to claim 1, wherein the user management unit assigns a higher authority attribute to a user holding the operation pen than to a user not holding the operation pen.

Images