JP2008112360A - User interface device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a new user interface device, and to improve convenience for a user. <P>SOLUTION: A noticing site specification part specifies a nose region in a recognized face region as a site under consideration (S118). A tracking part traces the position of the nose region of the acquired i-th frame by matching using a template image (S122). The tracking part tries the detection of the change of the direction of the face based on the tracing result (S126). When the change of the direction of the face of the user is detected in the i-th frame by the tracking part (YES in S128), an input processing part performs the input processing corresponding to meaning content showing the detected change of the direction of the face of the user by referring to the table (S130). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a user interface technique using image input.

For example, in game devices, it is widely used to operate a controller by operating up / down / left / right direction keys or an analog stick of a controller (Patent Document 1). Recently, technological innovations have made it possible to play games by shaking or rotating the controller.
JP 2004-313492 A

  The present inventor has recognized the following problems with current controllers. That is, for example, if the pointer is moved by the operation of the direction key, it takes time to move, and it cannot be said that it is convenient. Also, the analog stick may be difficult to operate due to the force of returning to the center. Furthermore, the operation of shaking or rotating the controller may be felt lonely by one person, or it may be difficult to continue for a long time due to fatigue. Thus, the current controller has room for improvement in terms of user convenience.

  The present invention has been made in recognition of such a situation, and an object of the present invention is to provide a new user interface device and improve convenience for the user.

  An aspect of the present invention relates to a user interface device, which estimates the orientation of a user's face based on a user's face image and executes a predetermined input process according to the estimation result. In another aspect, a predetermined input process is executed by paying attention to a change in the face orientation instead of the face orientation. In addition, what converted these aspects between the method or system, a computer program, a recording medium, etc. is also effective as this invention.

  According to the user interface device of the present invention, convenience for the user can be enhanced.

Before describing the embodiment in detail, an outline thereof is shown below.
1. A user interface device according to an aspect includes an acquisition unit that acquires a photographed user's face image, an estimation unit that estimates a user's face direction based on the acquired face image, and an estimated face direction And an input processing unit that executes predetermined input processing according to the above. Here, the “predetermined input process” means that the intention of the user is reflected in some form on the apparatus, and does not necessarily need to be accompanied by an input determining operation.

  2. In another aspect, an acquisition unit that sequentially acquires a user's face image taken a plurality of times, an estimation unit that sequentially estimates a user's face direction based on the acquired face image, and And an input processing unit that executes predetermined input processing in accordance with the change in the face orientation. The apparatus further includes a table describing the relationship between the change in face orientation and the input processing according to the meaning content, and the input processing unit refers to the table to indicate the change in face orientation. You may perform the input process according to the content.

  3. In yet another aspect, an acquisition unit that sequentially acquires a user's face image photographed multiple times, and a position of a target region of the face image acquired for the first time using a positional relationship of the region in the face image. By tracking the position of the target region of the face image acquired for the second time or later by matching using the template image, using the target region specifying unit to be identified and the partial image including the specified target region as the template image The tracking unit that detects the change in the orientation of the face, a table that describes the relationship between the change in the orientation of the face and the input processing according to the meaning content, and the meaning that the change in the orientation of the face represents by referring to the table An input processing unit that executes input processing according to the content. Here, the “first time” is not necessarily limited to the first shooting. To the last, it is sufficient that the imaging is used for specifying the region of interest first, and it is only called the first time because it is performed earlier than the subsequent tracking processing.

  Each aspect pays attention to the characteristic peculiar to the face. That is, for example, with a hand, the shape and the position change variously depending on time and circumstances, so that it is difficult to capture in image recognition. In addition, it is difficult to define the “direction” due to the wide movable range. On the other hand, if it is a face, the change in shape is relatively small and the movable range is limited. Therefore, the position can be easily identified by image recognition, and the “direction” can be defined. The present inventor has noticed such a characteristic unique to the face, and has come up with a technique for reflecting the orientation of the face or a change thereof in the input process in the user interface device.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In addition, the same code | symbol is attached | subjected to the same or equivalent component, member, and process which are shown by each drawing, and the overlapping description is abbreviate | omitted suitably. In addition, the embodiments do not limit the invention but are exemplifications, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

  FIG. 1 is a conceptual diagram illustrating a use environment of an information processing system 1 using a user interface device according to an embodiment. The function of this user interface device is realized by a computer 10 constituting a part of the information processing system 1. The information processing system 1 includes a digital camera 2 and a display 3 in addition to the user interface device 100.

  The digital camera 2 is provided at a position where the face image of the user in front of the display 3 can be taken. FIG. 1 illustrates the case where the digital camera 2 is provided on the display 3. Note that the image captured by the digital camera 2 may be a moving image having a predetermined frame rate or may be a series of still images. In the present embodiment, it is assumed that it is a moving image. The display 3 is not particularly limited, and displays, for example, a software keyboard or a menu screen according to the control of the computer 10. As will be described later, the computer 10 executes input processing according to an image captured by the digital camera 2, that is, a user's face image.

  FIG. 2 is a functional block diagram of the user interface device 100 implemented in the computer 10 shown in FIG. Each block shown here is realized in hardware by an element such as a CPU of a computer or a mechanical device, and in software it is realized by a computer program or the like. Draw functional blocks. Those skilled in the art will understand that these functional blocks can be realized in various forms by a combination of hardware and software.

  The user interface device 100 includes a buffer 11, an acquisition unit 12, an estimation unit 14, an input processing unit 16, and a table 18. The buffer 11 temporarily stores the captured image of the digital camera 2 for each frame by a known method. The acquisition unit 12 sequentially reads out the images stored in the buffer 11 for each frame for processing by the estimation unit 14. The estimation unit 14 includes a face recognition unit 20, an attention site specifying unit 22, and a tracking unit 24.

  The face recognition unit 20 recognizes an image area corresponding to the user's face in the acquired image of the acquisition unit 12. Hereinafter, an image area corresponding to a face in the acquired image is referred to as a “face area”, and an image including the face area is referred to as a “face image”. For the recognition of the face area, a known pattern recognition technique such as boosting or support vector machine may be used. For example, a reference image obtained by synthesizing several tens to several hundreds of face images is prepared in advance, and the face recognition unit 20 sets a rectangular area in the acquired image, and compares the reference image with the image of the rectangular area. Then, the similarity as an image is digitized. By setting various rectangular areas in the acquired image and calculating the similarity, the “face-like” area in the acquired image is specified as the face area. Note that the face recognition unit 20 recognizes the face area of the next acquired image if the face area cannot be recognized in the acquired image of the acquisition unit 12. These processes are repeated until the face area is recognized.

  The attention site specifying unit 22 specifies the attention site in the face area recognized by the face recognition unit 20. In the present embodiment, a region corresponding to the nose (hereinafter referred to as “nose region”) in the face region is recognized and specified as a target region. The recognition of the nose region may be the same as in the case of the face region, or the central part of the face region may be simply recognized as the nose region. Alternatively, the positions of both eyes that can be specified relatively easily may be specified first, and a predetermined range corresponding to the lower part of the center portion of the line connecting the eyes may be recognized as the nose region. That is, the nose region may be recognized using the positional relationship between the parts in the face image. FIG. 3 illustrates the nose region. A rectangular area surrounded by a broken line is a nose area. Unlike the eyes and mouth, the nose does not close or open, so in this embodiment, the nose region is specified as the site of interest. This facilitates tracking by the tracking unit 24.

  Returning to FIG. 2, the tracking unit 24 performs the following operation on each frame (hereinafter referred to as “subsequent frame”) after the frame (hereinafter referred to as “initial frame”) in which the target region, that is, the nose region is specified by the target region specifying unit 22. By tracking the nose area, the direction of the user's face is sequentially estimated and the change is detected. In this embodiment, the “face orientation” is based on the face when the user's face is facing the digital camera 2. Known template matching may be used for tracking the nose region. Specifically, the partial image including the nose region in the first frame is used as a template image, and the position of the nose region in the subsequent frame is tracked by matching using the template image. Here, it is not necessary to perform matching for the entire region of the subsequent frame, and it is sufficient to perform matching only for the peripheral portion of the nose region in the first frame based on the movable range of the face. The template image may be sequentially updated instead of using the partial image including the nose region in the first frame as a template image consistently.

As described above, the estimation unit 14 sequentially estimates the orientation of the user's face based on the user's face images sequentially acquired by the acquisition unit 12, and detects the change.
The table 18 describes the relationship between the change in the orientation of the face and the input processing according to the meaning content. The input processing unit 16 refers to the table 18 and executes input processing according to the meaning content represented by the change in the face orientation detected by the tracking unit 24. The result of the input process is output to the display 3. The input processing unit 16 may also perform input preparation processing such as return from the screen saver and menu display, using the fact that the face recognition unit 20 has recognized the face area as a switch.

Some description examples of the table 18, that is, some examples of input processing according to changes in the orientation of the face and the meaning contents thereof will be shown.
(1) Movement of pointer: Input processing is executed so as to move the pointer displayed on the display 3 in accordance with the change in the orientation of the face. For example, when the orientation of the user's face changes to the right, the input process is executed so that the pointer displayed on the display 3 moves to the right as viewed from the user. The same applies to changes in other directions. Since such input processing is considered to easily match the user's intuition, it is expected to improve convenience for the user. The character may be moved instead of the pointer.
(2) Selection of “Yes” or “No”: Generally, when expressing feelings such as consent or consent, a person shakes his head vertically. On the other hand, when expressing feelings such as disapproval or disagreement, a person shakes his head. Paying attention to this point, when the user's face orientation changes downward and returns to the original state, input processing when the user selects “Yes” is executed, and the face orientation changes to the left and right. When the process returns to, input processing when the user selects “No” is executed. According to this, in response to the “Yes” and “No” questions displayed on the display 3, the user crawls and shakes his / her head in the same manner as in normal conversation, and the operation is as intended. Input processing can be obtained. Therefore, compared with the case where the pointer is operated with a mouse or the like and “Yes” or “No” is selected, the input can be advanced with a more intuitive operation, and thus the convenience for the user is expected to be improved.
(3) Scrolling of screen: Input processing is executed so as to scroll the display on the display 3 in accordance with the change in the orientation of the face. For example, when the orientation of the user's face changes downward, the input process is executed so that the display 3 is scrolled downward. The same applies to changes in other directions. Scrolling may seem difficult for users who are not familiar with computer operations, but as described above, screen scrolling can be made more intuitive by associating changes in face orientation with screen scrolling orientation. Improvement of convenience for the user is expected.

  FIG. 4 is a flowchart showing processing in the user interface apparatus 100 shown in FIG. When the process is started, the frame count value i is set to 0 (S101). The acquisition unit 12 acquires the i-th frame from the buffer 11 (S102). The face recognition unit 20 tries to recognize the face area for the acquired i-th frame (S104). When the face area cannot be recognized (S106: No), i is incremented (S108), and the same processing is executed. These processes are repeated until the face area is recognized.

  When the face recognition unit 20 recognizes a face region in the i-th frame (S106: Yes), the attention site specifying unit 22 specifies the nose region in the recognized face region as the attention site (S118). The partial image including the identified nose region is held as a template image in the tracking unit 24.

  After specifying the nose region, i is incremented (S119), and the acquisition unit 12 acquires the i-th frame from the buffer 11 (S120). The tracking unit 24 attempts to track the position of the nose region for the acquired frame by matching using the template image (S122). If the nose area is lost and tracking is not possible (S123: No), i is incremented (S124), and the process returns to the step of S102. Whether or not the nose region can be tracked can be determined by the magnitude relationship between the template matching score and the threshold.

  When the tracking unit 24 succeeds in tracking the nose region (S123: Yes), the tracking unit 24 tries to detect a change in the orientation of the face based on the tracking result (S126). If no change in face orientation is detected (S128: No), i is incremented (S119), and the same processing is executed. These processes are repeated until a change in face orientation is detected.

  When the tracking unit 24 detects a change in the orientation of the user's face in the i-th frame (S128: Yes), the input processing unit 16 refers to the table 18 to determine the detected orientation of the user's face. An input process corresponding to the meaning content represented by the change is executed (S130).

  FIG. 4 shows a case where the tracking unit 24 attempts to detect a change in face orientation every time one frame is tracked. Instead, the tracking unit 24 changes the number of tracked frames until a predetermined number is reached. Tracking may be repeated, and detection of a change in face orientation may be attempted when the number of tracked frames reaches a predetermined number. The number of tracked frames when the detection is attempted (hereinafter referred to as “necessary number of frames”) may be set for each input process to be executed in the input processing unit 16.

  For example, when the input processing unit 16 performs an input process such as selecting “Yes” or “No”, the required number of frames is set to be larger. This is because, in order to select “Yes” or “No”, the tracking unit 24 must detect a change in at least two directions of the user's face direction. On the other hand, when the input processing unit 16 performs input processing such as pointer movement or screen scrolling, the required number of frames is set to be small. In this case, it is sufficient to detect a change in the direction of the face in one direction.

(Situation 1: When you cannot use your hands)
A case where the information processing system 1 shown in FIG. 1 is used in a terminal of an automobile factory will be described. Consider a scene in which a mechanic repairing a car in an automobile factory operates a computer in the factory to refer to parts information. In such situations, the hands are often dirty or wet and the computer cannot be operated by hand. According to the present embodiment, when the mechanic makes the face face the digital camera 2, the computer 10 displays a menu on the display 3 using it as a switch. The mechanic can refer to the parts information simply by changing the face orientation. Therefore, it is not necessary to wash hands in order to refer to the part information, and the waste of time for the mechanic can be suppressed.

(Usage scenario 2: Software keyboard)
A case where input processing using a software keyboard is executed in the information processing system 1 shown in FIG. 1 will be described. If you use direction keys or joysticks to identify characters on a software keyboard as in the past, you will have to stop moving each time you move a character, or you may overshoot the desired character and be easy to operate. I can't say that. Further, when an analog stick is used, the force to return to the middle may deteriorate the operability. According to the present embodiment, the user can move the pointer in an analog manner like a mouse by changing the direction of the face, which is convenient. Furthermore, space saving is expected in that a mouse and a mouse pad are not required.

(Usage scenario 3: selection of “Yes” or “No”)
A case where the information processing system 1 shown in FIG. 1 is used as an entertainment system will be described. Consider a scene where a user is playing an RPG (Role-Playing Game). Conventionally, a direction key, an analog stick, or the like is used to move a character, and when a “Yes” or “No” selection is made, the ○ button or the X button is pressed. In the present embodiment, the user can walk through the screen by moving the character by changing the orientation of the face. In addition, when speaking to another character and selecting “Yes” or “No”, the user can make a selection by whispering or shaking his / her head. Here, if the input processing unit 16 detects that the user has not responded for a predetermined time, the input processing unit 16 can naturally proceed with the game as if there was no selection.

  The above-described embodiments are merely examples, and those skilled in the art will readily understand that various modifications thereof are included in the present invention. Hereinafter, such a modification technique will be exemplified.

  Although the embodiment mainly focuses on the change of the face direction, the deformation technique focuses on the face direction itself. The tracking unit 24 does the same as the embodiment until the position of the nose region in the subsequent frame is tracked, and then determines the current face orientation from the position of the nose region in the first frame and the position of the nose region in the current frame. presume. Depending on the application, it may be more convenient to use the face orientation itself for input processing than to change the face orientation. In such a case, this modification technique is advantageous.

  In the embodiment, the case where the photographed image of the digital camera 2 is mainly a moving image has been described, but it may be a still image instead of a moving image. The estimation unit 14 can estimate the face direction based on the position of the nose region in the still image taken for the first time in the subsequent still image. In addition, the still image photographed for the first time means a still image in which the nose region is specified by the attention site specifying unit 22.

  In the embodiment, the case has been described where the attention site specifying unit 22 specifies the nose region of the face image as the attention site. In the deformation technique, a region corresponding to the eyes or mouth may be set as the attention site. Further, two or three of the positional relationship between eyes, mouth and nose may be set as the attention site. In the case of the eyes and mouth, the tracking unit 24 is not as easy to track as the nose, but by overcoming it, the same effects as in the embodiment can be obtained.

  Although not particularly mentioned in the embodiment, the user may change the position of the face without changing the orientation of the face. This is the case when the face is moved in parallel with the display surface of the display 3. Also in such a case, the user interface device 100 estimates that the face orientation has changed, and executes an input process corresponding to the change. For example, when the user moves the face to the left in parallel with the display surface of the display 3, the user interface device 100 estimates that the orientation of the user's face has changed to the left, and moves the pointer to the left. Input processing like this is executed. This input process is considered to be consistent with the user's intuition, and as a result can be said to be appropriate.

It is a conceptual diagram which illustrates the use environment of the information processing system using the user interface device concerning an embodiment. It is a functional block diagram of the user interface apparatus implement | achieved in the computer shown by FIG. It is explanatory drawing which overlaps and shows the acquired image by the acquisition part shown by FIG. 2, and the nose area recognized in an attention site | part specific part. It is a flowchart which shows the process in the user interface apparatus shown by FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Information processing system, 2 ... Digital camera, 3 ... Display, 10 ... Computer, 12 ... Acquisition part, 14 ... Estimation part, 16 ... Input processing part, 18 ... Table, 20 ... Face recognition unit, 22 ... Part of interest specifying unit, 24 ... Tracking unit, 100 ... User interface device.

Claims (4)

An acquisition unit for acquiring a photographed face image of the user;
An estimation unit that estimates the orientation of the user's face based on the acquired face image;
An input processing unit that executes predetermined input processing according to the estimated face orientation;
A user interface device comprising: An acquisition unit for sequentially acquiring a user's face image taken multiple times;
An estimation unit that sequentially estimates the orientation of the user's face based on the acquired face image;
An input processing unit that executes a predetermined input process in accordance with a sequentially estimated change in face orientation;
A user interface device comprising: The apparatus according to claim 2, further comprising a table that describes a relationship between a change in face orientation and input processing in accordance with the meaning content thereof.
The user interface device, wherein the input processing unit executes an input process according to a semantic content represented by a change in face orientation by referring to the table. A function of sequentially estimating the orientation of the user's face based on the user's face image taken multiple times;
A function of executing a predetermined input process in accordance with the sequentially estimated change in face orientation;
A user interface processing program characterized by causing a computer to exhibit the above.

Images