JP2000075991A - Information input device - Google Patents

Abstract

(57) [Abstract] [Problem] Without seeing or touching onboard equipment,
Provided is a method of inputting information in which an information input operation can be performed in a non-contact manner while maintaining a driving posture and input information can be easily confirmed. A driver raises a hand so as to enter a range of a virtual space, and changes a hand from a closed state to an open state. This hand movement is imaged by the camera, and when the image and the position correspond to predetermined ones, a standby state where input is possible is established. Subsequently, the driver 54 grasps one of the menu spaces MA to MD in the virtual space 50 to be selected with the hand 52. This operation is similarly imaged by the camera, the movement and position of the hand 52 are recognized, and it is determined which menu space the hand 52 has grasped. The determination result is supplied to the navigation device. The judgment result is called back to the driver 54 by voice.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information input device for inputting information such as commands to various devices.
For example, the present invention relates to an information input device suitable for a vehicle-mounted device such as a car navigation system.

[0002]

2. Description of the Related Art In car navigation systems, various services such as navigation for road guidance and the like, video display of television and video software, audio playback of radio and CD playback, execution of game software playback, control of equipment such as air conditioners, etc. Is possible. These service menus are usually hierarchized, and the user selects a necessary one from the hierarchized menu.

FIG. 10 shows an example of such a hierarchical menu. In this example, as shown in FIG. 7A, four main menus MN1 to MN4 of "search", "see", "remember", and "change" are displayed on a touch panel which is a display of the navigator device. .
Among them, for example, the menu of “search” is hierarchized into menus 1, 2, 3, 4,... As shown in FIG. Further, the menu 1 is the menu 11, 1
It is hierarchized into 2, 13, 14, ..., and menu 1
1 is a menu 111, 112, 113, 114,...
It is hierarchically arranged. The same applies to other menus.

[0004] By the way, such menu selection is performed by:
If a menu selection icon is displayed on the display with the touch panel, touch the icon. If there is a menu selection switch button, operate it. If there is a remote control (remote control) device, touch it. It is done in such a way as to operate.

[0005]

However, the icons for menu selection displayed on the touch panel screen are:
Since the display position relatively fluctuates according to the display content, it is difficult to grasp the display position. For this reason, it is not always easy to perform an icon operation on the touch panel screen. In addition, in order to operate the touch panel and the switch buttons, it is necessary to reach to those positions, and the driver must break the driving posture even temporarily. In the case of a remote control device, the user must search for and operate the remote control itself. Further, in any case, it is necessary to confirm whether a necessary menu is selected by looking at the screen.

However, from the viewpoint of driving a car more comfortably, a driver can select a necessary menu without moving his / her gaze toward the display screen or changing his / her posture to extend his / her arm. It is convenient to be able to.

[0007] The present invention focuses on the above points, and its object is to provide a vehicle-mounted device without looking or touching it.
An object of the present invention is to provide an information input device capable of performing an information input operation without contact. Another object is to provide an information input device capable of performing an information input operation while maintaining a driving posture. Still another object is to provide an information input device with good operability that can easily confirm input information.

[0008]

In order to achieve the above object, the present invention provides an image pickup means for picking up an image of a part of a human body; at least the shape, movement, or position of an object from an image picked up by the image pickup means. Standby determining means for grasping one and determining a standby state of information input; after determining that the apparatus is in the standby state, the image captured by the image capturing means is used to determine the shape, movement,
Alternatively, there is provided an input determining means for determining at least one of the positions to determine the input information; and a callback means for calling back a determination result by the standby determining means and the input determining means.

One of the main modes is characterized in that the object to be imaged is a hand of a human body, and the imaging means images the object in a predetermined virtual space. In another aspect, the position of the symmetric object is a two-dimensional or three-dimensional coordinate position in the virtual space.
Further, the image pickup means performs the image pickup with the ceiling of the automobile as a background. Still another mode is characterized in that after the callback by the callback means based on the determination result of the input determination means, the input determination means determines the input confirmation.

According to another aspect of the present invention, there is provided an in-vehicle device operating device for operating an in-vehicle device, comprising: an image pickup device; a part of a human body image picked up by the image pickup device; Recognizing means for recognizing a part of the movement of the human body as a specific change pattern; storage means for storing the specific change pattern in association with the operation information of the vehicle-mounted device;
On-board equipment operating means for reading out from the storage means operation information corresponding to the specific change pattern recognized by the recognition means, and operating on-vehicle equipment based on the read operation information. Features.

According to the present invention, the operation of the on-vehicle equipment can be performed only by the movement of a part of the human body, such as the movement of the hand. The on-vehicle devices operated by the present invention include so-called car navigation devices for performing route guidance and the like, air conditioners, televisions, radios, audio devices such as cassettes and CDs, wipers, headlights, and various other on-vehicle devices. It is.

[0012] The operations of the on-vehicle devices include all operations performed by the user on the on-vehicle devices. For example, power on / off operation of various in-vehicle devices,
Operations such as destination setting and guidance start of car navigation system, operations such as air conditioner temperature control, air volume control, cooling / heating switching, TV / radio volume control / channel switching, cassette / CD volume control / playback stop Switching operation, on / off switching operation of wiper / headlight,
Other various operations are included.

The movement of a part of the human body is not limited to two-dimensional or three-dimensional movement, such as moving the hand up, down, left, right, back and forth, as well as when holding an open hand. It also includes movements that change shape. What is two-dimensional movement?
The movement in a plane including the predetermined XY axis is referred to as the three-dimensional movement. The three-dimensional movement is obtained by adding the movement in the Z-axis direction orthogonal to the plane including the predetermined XY axis to the two-dimensional movement. Say.

[0014] If the operation of switching the enlargement / reduction of the map display of the car navigation apparatus is performed by three-dimensional movement of the hand, the operation of switching the map can be performed more intuitively.
For example, moving the hand in the Z-axis direction
A reduction operation is performed, and another operation is performed by moving a hand in the XY plane. In this case, if the Z axis is set to extend in the front direction of the human body, the map can be switched more intuitively.

Further, in order to achieve the above object, the present invention is characterized in that it comprises a speaker; and a callback means for outputting a voice indicating the recognition result by the recognition means from the speaker. According to the present invention, since the user can confirm how the hand movement is recognized by voice, the user does not need to move his / her eyes for confirmation, and can drive more comfortably. it can. For example, when it is recognized that the hand has been swung right and left, a sound is output by the callback means as "recognition of right and left has been recognized".

In order to achieve the above object, the present invention further comprises a speaker; callback means for outputting a sound indicating the operation content from the speaker before operation by the onboard equipment operation means. It is characterized by. ADVANTAGE OF THE INVENTION According to this invention, a user can confirm by audio | voice how the onboard equipment is operated before operating onboard equipment. For this reason, it is not necessary to move the line of sight for the confirmation, and driving can be performed more comfortably. For example, a sound is output as “turn off the navigation device”.

Further, in order to achieve the above object, the present invention provides an image display device; display means for displaying on the image display an image showing a partial movement of a human body recognizable by the recognizing means. It is characterized by having. According to the present invention, it is clear how to move a hand to perform a specific operation of the vehicle-mounted device. The image displayed by the display means is, for example, an image indicating that when the vehicle-mounted device is an air conditioner, moving the hand to the left decreases the temperature, and moving the hand to the right increases the temperature. . This image may be a text-only display, or a right-arrow or left-arrow image together with the text.

The above and other objects, features, and advantages of the present invention will be apparent from the following detailed description and the accompanying drawings.

[0019]

Embodiments of the present invention will be described below in detail. FIG. 1 shows a configuration of one embodiment. First, from the information input device 10 side, the output side of the camera 11 which is an imaging unit is connected to the image processing unit 12. The output side of the image processing unit 12 is connected to the image recognition unit 14 and the position measurement unit 16, and the output sides of the image recognition unit 14 and the position measurement unit 16 are connected to the standby determination unit 18 and the input determination unit 20. Each is connected. The standby determination unit 18 and the input determination unit 20 each include a memory 18M storing information for determination,
20M are provided.

The standby determining unit 18 and the input determining unit 20
Is output to the main computer 3 of the navigation device 30 via appropriate interfaces 22 and 24.
2 respectively. The navigation device 30 as a vehicle-mounted device has a display unit 34 for performing display processing,
An audio unit 36 for performing audio processing and a navigation device 38 for a navigator are provided. The display unit 34 includes a display 34 as a display unit having a touch panel.
A, and the audio unit 36 includes a speaker 36A as audio output means. Main computer 32
Is a display control unit 32 for driving and controlling those devices.
A, a voice control unit 32B and a navigation control unit 32C are functionally provided.

Of the above components, the camera 11 is provided at a position below the dashboard of the automobile 40, for example, as shown in FIG. 2, and the imaging range is set so that the ceiling 42 of the automobile 40 has a field of view. ing. As described later, in the present embodiment, the shape and movement (gesture) of the driver's hand are captured by the camera 11. At this time, by setting the background of the hand to be the ceiling 42, reflection of unnecessary images can be reduced.

The image processing section 12 performs a process of removing a background from an image picked up by the camera 11 and extracting an image portion of a hand. The image recognition unit 14
This is for recognizing the image of the driver's hand. For example, the shape of a hand, such as a hand being open or a hand being held, is recognized by pattern recognition using a template. The position measuring unit 16 is for detecting the position of the image in a predetermined space from the center of gravity of the image, for example.

The patterning of the hand movement is performed, for example, as follows. The movement of the hand is detected by the position measuring unit 16 at a certain time interval, represented by continuous coordinate values, and stored in a memory (not shown). This continuous coordinate value is associated with a specific change pattern stored in a memory (not shown) and a representative coordinate value indicating the change pattern (a representative number of coordinate values among the continuous coordinate values). By comparing the table with the table, the image recognition unit 14 as a recognition unit recognizes a specific change pattern.

The standby judging section 18 is provided with the image recognizing section 14
It is determined whether the hand has a predetermined shape or changes in a predetermined virtual space based on the shape information of the hand recognized in the above and the position information measured by the position measurement unit 16. It is for. Hand shape information and position information necessary for the determination are set in advance and stored in the memory 18M. For example, the predetermined change of the hand is determined by converting a preset change pattern into a table and comparing the table with the pattern. The virtual space 50 is virtually set within the imaging range of the camera 11, as shown in FIG. It is determined whether or not there is a hand 52 in the virtual space 50, and the information input device 10 enters a standby state where input is possible.

The input judging section 20 judges input information based on the image information of the hand recognized by the image recognizing section 14 and the position information measured by the position measuring section 16. Information on the hand shape and position required for judgment is
It is set in advance and stored in the memory 20M as storage means. Specifically, as shown in FIG. 5B described later, the menu area 50A of the virtual space 50
Four menu spaces MA to MD of “search”, “see”, “remember”, and “change” in FIG. 10A are virtually set. Then, it is determined which menu has been input from the change in the shape and position of the hand, in which one of these menu spaces is grasped with the hand and moved to the work area 50B.

If the specific change pattern and the operation information of the navigation device 30 are stored in the memory 20M in association with each other, the input determining unit 20 can recognize the specific change pattern recognized by the image recognizing unit 14. , The operation information corresponding to the change pattern can be read from the memory 20M. The read operation information is sent to the main computer 32, and the navigation device 30 is operated based on the operation information.

Next, the main computer 32 of the navigation device 30 controls the operation of the navigation device 30 as a whole. Navigation device 30
The display 34A having a touch panel, which is a main device of, is for displaying maps, television, video software, menus, and the like. The speaker 36A is for performing voice navigation, radio and CD playback, and the like. The navigation device 38 includes a GPS receiver and a drive device such as a disk in which map information is stored, and performs a road display near a current position, a route search to a destination, a voice guidance, a display of VICS information, and the like. . The operations of these units are controlled by the control units 32A to 32C of the main computer 32.

Next, the operation of this embodiment will be described. FIG. 4 shows the main procedure of the operation as a flowchart.
FIGS. 5, 6, and 8 show hand movements in main operations. In the operation described below, it is assumed that the driver is driving and the information input device 10 and the navigation device 30 are operating.

First, the information input device 10 is in the reset state S
A, the camera 11 is in an imaging state. Here, when it is desired to input information such as changing the operation being executed by the navigation device 30, the driver raises his hand 52 so as to enter the range of the virtual space 50 as shown in FIG. When viewed from the front, the driver 54 raises his left hand 52 as shown in FIG. Then, the driver 54 makes the hand 52 have a predetermined shape or performs a predetermined shape change. For example, the state is changed from a state in which the hand is closed to a state in which the hand is opened.

The shape and change of the hand are captured by the camera 11 (step SB in FIG. 3), and after the processing by the image processing unit 12, the image recognition unit 14 recognizes the shape and change. The position measuring unit 16 measures the positions X and Y of the hand 52 in the virtual space 50 (Step SB).
These image recognition results and position measurement results are supplied to the standby determination unit 18. The standby determination unit 18 uses the data stored in the memory 18M to
Is determined in the virtual space 50 to have a predetermined shape or change. As a result, when the information corresponds to a predetermined shape or the like (Y in step SC), it is determined that the information input by the information input device 10 will be performed subsequently, and the fact is output to the main computer 32 of the navigation device 30. In response to the input, the main computer 32 notifies by voice that the apparatus is in the standby state (step S).
D). That is, the speaker 3 is controlled by the audio control unit 32B.
6A is driven, and a callback, for example, "Standby OK" or "Input is possible" is performed. Note that a callback may be made with an appropriate sound.

By such a callback, the driver 54 can know that the movement of the hand 52 has been recognized by the navigation device 30 without looking at the display 34A. At this time, the shape and position of the hand 52 may be displayed on the display 34A. In this way, for example, a passenger in the passenger seat can be notified that an input is made by the driver, and the driver can use the driver himself to confirm standby if the signal is stopped. The image recognition unit 14
34A displays an image showing hand movements that can be recognized by the user.
May be displayed.

In such a standby state, the driver 5
4, as shown in FIG. 5B, the user 52 grasps a desired one of the menu spaces MA to MD of the menu area 50 </ b> A virtually set in the virtual space 50 with the hand 52. Although the menu is shown in FIG. 5B for convenience, it is actually a space, and the driver 54 grasps the space. The shape and change of the hand 52 in this operation are similarly imaged by the camera 11 and
After the processing in step 2, the image is recognized by the image recognition unit 14 (step SE). The recognized hand shape and the like are supplied to the input determination unit 20 side under the control of the main computer 32. The input determination unit 20 determines which menu space the hand 52 has grabbed, using the data stored in the memory 20M.

As a result, when the grasped menu is determined (step SF), the fact is notified to the navigation device 3.
0 to the main computer 32. The main computer 32 notifies the grasped menu by voice according to the input (step SG). For example, a callback ““ Search ”has been selected” is performed. The driver 54 can confirm the menu selected by this callback.

If the selected menu is good, the hand 52 is moved from the menu area 50A to the work area 50B as a confirmation operation. When this is determined by the input determination unit 20 (Y in step SH), the menu command is input to the main computer 32 (step SI).
By such a confirming operation, the driver 54 can inform the device that the driver has selected the corresponding menu. If the menu called back is different, no confirmation operation is performed (step S
H) of N), the operation of grasping any menu space with the hand 52 again is repeated. As described above, one of the four menus of "search", "see", "remember", and "change" is selected.

By the way, as shown in FIG.
The menu "Search" has a hierarchical structure, and there is a lower menu. The driver 54 selects a further necessary one from among them. The operation at this time is
For example, this is performed as shown in FIG. First, in FIG. 6 (A), by shaking or moving the hand 52 tied to the “search” menu space MA in FIG. 5 (B) in the left-right direction, any one of the first hierarchical menus 1 to 4 is displayed. Select The corresponding first-level menu is called back according to the position of the hand 52 in the left-right direction. The driver 54 hears the callback and selects a necessary menu. That is, the hand 52
When the user shakes or moves right and left, the menu icon at the corresponding position becomes active. The menu selection is confirmed by, for example, an operation of once opening and closing the hand after hearing the callback of the corresponding menu.

FIG. 6B shows an example in which one of the second hierarchical menus is selected by moving the hand up and down. For example, menu 1 as the first hierarchical menu
Is selected. This menu 1 contains
As shown in (B), menus 11 to 14 also exist as the second hierarchy. Therefore, the driver 54 moves the hand 52 up and down to select a necessary second hierarchical menu. The corresponding second hierarchical menu is called back in accordance with the position of the hand 52 in the vertical direction.
Listen to it and select the required menu. That is, when the hand 52 is shaken or moved in the vertical direction, the menu icon at the corresponding position becomes active. Confirmation of menu selection is the same as in the case of the first hierarchy.

By repeatedly performing the above operation, the driver 54 can select a desired menu. Then, when the lowest menu is selected, the command corresponding to that menu is executed. In the operation shown in FIG. 4, whether all the inputs of the driver 54 are completed (step S
J, Y), the process ends when the driver 54 inputs an end command. The end command includes, for example, opening and shaking the hand in the virtual space 50. Thereafter, the driver 54 returns the hand 52 to a position such as a handle, and the driver 5
When the input determination unit 20 determines that the fourth hand 52 goes out of the virtual space 50 and there is no hand in the virtual space, the information input apparatus 10 is again in a state of waiting for input (step SA). . That is, the state returns to the state before the standby.

According to the present embodiment, the following effects can be obtained. (1) It is only necessary to move a hand in a virtual space, and an information input operation can be performed in a non-contact manner without looking or touching a touch panel or the like. In addition, the information input operation can be performed while maintaining the driving posture. (2) Since the input information is called back, the input information can be easily confirmed and the operability is good. (3) Since the operation of confirming the input information is performed, inconvenience such as execution of an incorrect command can be avoided. (4) For example, the user can select a menu by grasping the menu space, open his hand when finishing work, and shake the hand. I have. Therefore, it is possible to input information without performing a special hand movement. (5) By setting the camera so that the background of the hand is on the ceiling of the car, the reflection of unnecessary images can be reduced.

Next, another embodiment of the present invention will be described. In the above embodiment, one camera is used as an image sensor, and two-dimensional directions in the virtual space 50 are used.
That is, the positions of the hands in the X and Y directions shown in FIG. 3 were measured.
However, in the following embodiment, two cameras 11A and 11B are used as shown in FIG.
The hand position is measured in the three-dimensional direction of Z. The measurement of the three-dimensional coordinates X, Y, Z by two cameras is possible, for example, based on the principle of triangulation. In each of the cameras 11A and 11B, unnecessary image reflection can be similarly reduced by setting the imaging field of view so that the ceiling 42 of the automobile 40 appears as a background as shown in FIG. .

The operation in this embodiment is basically the same as the above-described embodiment. However, the position measurement unit 16 measures positions in three directions of X, Y, and Z. For this reason, the standby determination unit 18 and the input determination unit 20 perform determination in consideration of the Z direction. That is, the standby determination unit 18 determines the standby based on the virtual space 60 extending in the Z direction. The input determination unit 20 performs input determination in consideration of the shape and change of the hand 52 in the Z direction.

According to the example shown in FIG. 10B, the first
Below the hierarchical menu 1, the second hierarchical menus 11 to 11 are further provided.
14, and a third sub-menu is located below the second-level menu 11.
There are hierarchical menus 111 to 114. In such a hierarchical structure, this embodiment is convenient when it is desired to quickly move up and down the hierarchy, for example, when it is desired to quickly move to the menu 111 after selecting the “Search” menu.

FIG. 8 shows an example of such a movement between the layers. By moving the hand 52 in the Z direction as shown in FIG. Will be For example, when the hand 52 is closest to the camera 11A, the first hierarchical menus 1 to 4 can be selected.
By pulling backward, the second level menus 11 to 14 can be selected, and by further pulling the hand backward, the third level menu 1 can be selected.
11 to 114 can be selected. It is as if there is a menu of a lower hierarchy in the drawer of a menu of an upper hierarchy. The menu selection at each level may be performed by, for example, moving the hand 52 up and down as shown in FIG. When the driver is familiar with the menu hierarchy in advance, it is possible to quickly select a menu by moving between such hierarchies.

The present invention has many embodiments, and can be variously modified based on the above disclosure. For example, the following is also included.

(1) In the above-described embodiment, the example of the movement such as grasping with the hand or opening the hand has been described. However, the shape or change for input may be appropriately set as needed. For example, if you hold one finger from the gripped state, it corresponds to the shape and movement of the finger, such as menu 1, 2 and menu 2, etc., the degree of opening of the hand, the rotation of the open hand, etc. like,
Make the size of the area of the hand to be imaged correspond to the menu,
And so on.

(2) In the above embodiment, the menu is selected and the corresponding operation is executed as a result. However, commands are directly input by directly associating hand shapes and movements with predetermined commands. It may be. For example, it is convenient to make it possible to directly input a command for canceling the input halfway. FIG. 9 shows an example in which a corresponding operation is directly executed based on the shape and movement of the hand. First, in the example of FIG. 7A, opening and closing of a hand corresponds to opening and closing of a menu window. Opening the hand 52 opens the menu ME. Alternatively, it may be possible to correspond to ON / OFF of a vehicle-mounted device such as an air conditioner or a radio. In the example of FIG. 7B, the movement of the hand 52 in the X and Y directions is made to correspond to the scroll of the map on the display 34A. FIG. 11C shows the movement of the hand 52 in the Z direction corresponding to the change in the scale of the map on the display 34A. This example is a radio or CD
It is also applicable to changing the volume of reproduction.

(3) The function of the information input device may be included in the navigation device. For example, image recognition, position measurement, standby, and input determination are performed by the main computer.

[0047]

As described above, according to the present invention,
The following effects are obtained. (1) It is possible to input information while maintaining a driving posture in a non-contact manner without looking at a display or touching a switch, thereby improving driving comfort. (2) By calling back the input information, the input information can be easily confirmed and the operability is good.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an information input device according to one embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of an arrangement of cameras in the device.

FIG. 3 is a diagram showing a virtual space for information input in the embodiment.

FIG. 4 is a flowchart showing a procedure of information input in the embodiment.

FIG. 5 is a view showing a movement of a driver's hand in the embodiment.

FIG. 6 is a diagram showing a hand movement of another driver in the embodiment.

FIG. 7 is a diagram showing a virtual space for inputting information in another embodiment.

FIG. 8 is a diagram showing a movement of a driver's hand in another embodiment.

FIG. 9 is a view showing movement of a driver's hand according to still another embodiment.

FIG. 10 is a diagram showing an example of a menu and its hierarchical structure.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Information input device 11, 11A, 11B ... Camera 12 ... Image processing part 14 ... Image recognition part 16 ... Position measurement part 18 ... Standby determination part 18M, 20M ... Memory 20 ... Input determination part 22, 24 ... Interface 30 ... Navigation Device 32A Display control unit 32B Voice control unit 32C Navigation control unit 32 Main computer 34 Display unit 34A Display with touch panel 36 Voice unit 36A Speaker 38 Navigation device 40 Vehicle 42 Ceiling 50, 60 ... Virtual space 50A ... Menu area 50B ... Work area 52 ... Hand 54 ... Driver MA-MD ... Menu space ME ... Menu MN1-MN4 ... Main menu

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Shoji Yokoyama 2-19-12 Sotokanda, Chiyoda-ku, Tokyo F-term in Equos Research Co., Ltd. 5H180 AA01 BB13 CC04 FF03 FF13 FF22 FF25 FF27 FF32 FF33

Claims (5)

[Claims] 1. An information input device for inputting information to an in-vehicle device, comprising: an image pickup unit for picking up an image of a part of a human body; Standby determining means for grasping one and determining a standby state of information input; from the image captured by the image capturing means after being determined to be in the standby state by this, the shape, movement, or position of the object An information input device, comprising: input determination means for determining at least one of the following to determine input information; callback means for calling back a determination result by the standby determination means and the input determination means. 2. The information input device according to claim 1, wherein the object to be imaged is a hand of a human body, and the imaging unit images the object in a virtual space set in advance. 3. The information input device according to claim 2, wherein the position of the symmetric object is a two-dimensional or three-dimensional coordinate position in the virtual space. 4. The information input device according to claim 1, wherein the image capturing means performs the image capturing such that a ceiling of the automobile is a background. 5. The information input device according to claim 1, wherein after the callback by the callback unit based on the determination result of the input determination unit, the input determination unit determines the input confirmation.