WO2025173525A1 - Information processing device, information processing method, and program - Google Patents

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program, that enable realizing a higher degree of freedom in operability. An information processing device according to the present invention estimates a finger and hand shape of a user and a force state of a hand of the user on the basis of myoelectric potential data corresponding to movement of the hand and the fingers of the user in a state of gripping or supporting a given real object, and executes processing corresponding to operations performed by the user. Further, the information processing device detects that the user has performed an initial operation, on the basis of the finger and hand shape of the user and the force state of the fingers of the user. The present technology can be applied to, for example, a terminal operation system using a given real object.

Description

Information processing device, information processing method, and program

This disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable greater operability.

Technology has been developed to measure the myoelectric potential of a user's arm and operate an information processing device in response to the movements of the user's hands and fingers.

For example, Patent Document 1 discloses a motion recognition method using a grasped object, which estimates the state of a user's wrist through a writing action using the grasped object, estimates the joint movements of the user's body parts associated with the wrist through the writing action, and estimates the state of the grasped object from the wrist state and joint movements.

JP 2015-001978 A

As disclosed in the aforementioned Patent Document 1, there is a demand not only for the recognition of content written using a pen-shaped, elongated object, but also for the realization of a high degree of freedom in operability, for example, by enabling the use of a variety of arbitrary objects.

This disclosure was made in light of these circumstances, and aims to enable greater operability with greater flexibility.

An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires measurement data corresponding to the movement of a user's hand and fingers while gripping or supporting an arbitrary real object, and a calculation unit that estimates the shape of the user's fingers and the force state of the hand based on the measurement data, and executes processing corresponding to the user's operation.

An information processing method or program according to one aspect of the present disclosure includes acquiring measurement data corresponding to the movement of a user's hand and fingers while gripping or supporting an arbitrary real object, estimating the shape of the user's fingers and the force state of the hand based on the measurement data, and executing processing corresponding to the user's operation.

In one aspect of the present disclosure, measurement data is acquired according to the movement of a user's hand and fingers while grasping or supporting any real object, and the shape of the user's fingers and the force state of the hand are inferred based on the measurement data, and processing is performed according to the user's operation.

1 is a block diagram showing a configuration example of an embodiment of a terminal operation system to which the present technology is applied. FIG. 2 is a block diagram showing an example of the configuration of a myoelectric potential measuring device and an information processing terminal. FIG. 10 is a diagram illustrating a recognition model used to identify the user's hand shape. 10A and 10B are diagrams illustrating the determination of a force state based on a threshold value. 10A and 10B are diagrams illustrating the identification of a force state using a recognition model. FIG. 10 is a diagram illustrating detection of an initial operation. FIG. 10 is a diagram illustrating calibration for setting a threshold value. 10A and 10B are diagrams illustrating hand and finger shapes, operation modes, and operation contents. 10A and 10B are diagrams illustrating hand and finger shapes, operation modes, and operation contents. 10 is a flowchart illustrating a myoelectric potential measurement process. 10 is a flowchart illustrating an operation execution process. FIG. 10 is a diagram illustrating an example of a usage example using a smart watch. FIG. 10 is a diagram illustrating an example of use when a user cooks. 10A and 10B are diagrams illustrating an example in which the display size of an operation guide is changed over time. FIG. 10 is a diagram illustrating an example of an operation on a keyboard. 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied.

Specific embodiments applying this technology will be described in detail below with reference to the drawings.

<Example of terminal operation system configuration>
FIG. 1 is a diagram showing an example of the configuration of an embodiment of a terminal operation system to which the present technology is applied.

As shown in Figure 1, the terminal operation system 11 is configured with an electromyography measuring device 12 and an information processing terminal 13.

The myoelectric potential measuring device 12 measures the myoelectric potential of the arm of a user using the terminal operation system 11, and communicates with the information processing terminal 13 to transmit the myoelectric potential data obtained as a result of measuring the user's myoelectric potential. While Figure 1 shows a ring-shaped myoelectric potential measuring device 12 worn on the user's right arm, the myoelectric potential measuring device 12 can be worn anywhere on the user's arm, from the shoulder to the wrist on either the left or right side.

For example, the myoelectric potential measuring device 12 may be configured with multiple myoelectric potential sensors that are in close contact with the skin surface, and each of the myoelectric potential sensors may have at least three electrodes: a positive electrode, a negative electrode, and a reference electrode. The electrodes of the myoelectric potential sensors may be configured so that the reference electrode is positioned on a straight line between the positive and negative electrodes, or so that the reference electrode is positioned elsewhere. The myoelectric potential measuring device 12 may also be configured so that multiple myoelectric potential sensors are arranged in a grid pattern.

The information processing terminal 13 is an electronic device that is operated by a user of the terminal operation system 11. In the example shown in Figure 1, a head-mounted display for AR (Augmented Reality) or VR (Virtual Reality) is illustrated as an example of the information processing terminal 13, but it may also be a smartphone, a smartwatch, or the like. The information processing terminal 13 communicates with the electromyography measuring device 12, receives electromyography data transmitted from the electromyography measuring device 12, and performs processing according to the user's operation on the information processing terminal 13 based on the electromyography data.

For example, when a user grasps or supports any real object 21 and applies force to multiple fingers to perform an initial movement, the myoelectric potential of the user's arm corresponding to the force applied to each finger is measured by the myoelectric potential measuring device 12, and the information processing terminal 13 detects that an initial movement has been performed based on the myoelectric potential data. Then, based on the myoelectric potential data, the information processing terminal 13 recognizes the shape of the user's fingers (the shape of the hand and fingers) and the force state of the user's fingers (which of the five fingers is tensing, the strength of the tension, etc.), and can determine the operation content of the operation performed by the user on the information processing terminal 13 based on the inferred result.

As a result, a user wearing the myoelectric potential measuring device 12 can grasp or support any real object 21 with a specified hand and finger shape, and perform various operations on the information processing terminal 13 by applying pressure with the desired fingers.

In the terminal operation system 11 shown in FIG. 1, the information processing terminal 13 that recognizes the user's hand shape and finger force state and performs processing to determine the operation content of the operation performed by the user is the target of the user's operation, but an electronic device other than the information processing terminal 13 that performs such processing may also be the target of the user's operation. In other words, the information processing terminal 13 can notify other electronic devices of the operation content, so that processing according to the user's operation can be performed.

Figure 2 is a block diagram showing an example configuration of the myoelectric potential measuring device 12 and the information processing terminal 13.

As shown in FIG. 2, the myoelectric potential measuring device 12 is configured with eight myoelectric potential sensors 31-1 to 31-8, a signal acquisition unit 32, a signal processing unit 33, and a communication unit 34, and the information processing terminal 13 is configured with a communication unit 41, a memory unit 42, and a calculation unit 43.

As shown in FIG. 1, when the myoelectric potential measuring device 12 is worn on the user's arm, the myoelectric potential sensors 31-1 to 31-8 are placed at predetermined intervals around the user's arm, and output myoelectric potential signals (electrical signals obtained by measuring and amplifying the potential difference between the positive electrode, negative electrode, and reference electrode) corresponding to the weak electric field generated in the muscles of the user's arm at each position. Note that, when there is no need to distinguish between the myoelectric potential sensors 31-1 to 31-8, they will hereinafter be simply referred to as myoelectric potential sensors 31. Furthermore, while the example shown in FIG. 2 uses eight myoelectric potential sensors 31-1 to 31-8, the number of myoelectric potential sensors 31 provided in the myoelectric potential measuring device 12 may be eight or more, or eight or less.

The signal acquisition unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 and supplies them to the signal processing unit 33.

The signal processing unit 33 performs signal processing such as noise removal and Fourier transform on the myoelectric potential signal supplied from the signal acquisition unit 32, for example, using a circuit that includes an operational amplifier and a high-pass filter, or through digital processing. The signal processing unit 33 then supplies the myoelectric potential data (a waveform that indicates changes in myoelectric potential in response to the movement of the user's hand and fingers) obtained as a result of the signal processing on the myoelectric potential signal to the communication unit 34.

The communication unit 34 communicates with the communication unit 41 of the information processing terminal 13 and transmits the myoelectric potential data supplied from the signal processing unit 33.

The communication unit 41 communicates with the communication unit 34 of the EMG measuring device 12 to receive EMG data and supplies it to the calculation unit 43. The communication unit 34 and the communication unit 41 can communicate using a wired cable, a wireless LAN (Local Area Network), Bluetooth (registered trademark), etc.

The memory unit 42 is a memory or storage that stores pre-registered data such as the operating form of the initial operation, the correspondence between the user's finger shapes and operation modes, and the operation content for each operation mode.

The calculation unit 43 is composed of a CPU (Central Processing Unit), IC (Integrated Circuit), etc., and uses machine learning models, recognition models, algorithms, etc. to perform calculations to estimate the user's hand shape and hand force state based on the myoelectric potential data supplied from the communication unit 41. The calculation unit 43 then obtains the operation mode corresponding to the user's hand shape from the registered data in the memory unit 42, and in that operation mode, performs processing in accordance with the operation content corresponding to the force state of the user's hand.

As shown in the figure, the calculation unit 43 is configured to include a hand shape identification unit 51, a force state identification unit 52, an initial movement detection unit 53, an operation mode selection unit 54, and an operation execution unit 55.

The hand posture identification unit 51 identifies the user's hand posture based on the myoelectric potential data supplied from the communication unit 41, and supplies the recognition result indicating the user's hand posture to the initial movement detection unit 53 and the operation mode selection unit 54.

For example, as shown in FIG. 3, myoelectric potential data for various hand and finger postures, such as an open hand, a hand gripping an object with the entire hand, and a hand pinching an object with certain fingers, can be obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8. The hand and finger posture identification unit 51 can then identify the user's hand and finger posture by using a recognition model generated by machine learning (e.g., k-nearest neighbor method, support vector machine, neural network, etc.) that uses this myoelectric potential data as input. In other words, the hand and finger posture identification unit 51 can input the myoelectric potential data supplied from the communication unit 41 into the recognition model and output a recognition result indicating the user's hand and finger posture recognized from the myoelectric potential data.

The force state identification unit 52 identifies the force state of the user's finger based on the myoelectric potential data supplied from the communication unit 41, and supplies the recognition result indicating the force state of the user's finger to the initial action detection unit 53 and the operation execution unit 55.

For example, as shown in FIG. 4, the force state determination unit 52 can determine the force state of the user's finger in accordance with a threshold value for the myoelectric potential data. That is, the force state of the user's finger is determined based on whether the peak value of the myoelectric potential data obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 is equal to or greater than a preset threshold value shown by the dashed dotted line.

Specifically, in the example shown in FIG. 4, if all of the peak values of the myoelectric potential data corresponding to myoelectric potential sensors 31-1 to 31-8 are below the threshold, the force state is identified as one in which the hand is spread out and stationary on the desk. If the peak values of the myoelectric potential data corresponding to myoelectric potential sensors 31-1, 31-3, and 31-6 to 31-8 among the myoelectric potential data corresponding to myoelectric potential sensors 31-1 to 31-8 are above the threshold, the force state is identified as one in which the hand is spread out on the desk and the index finger is tensed. If the peak values of the myoelectric potential data corresponding to myoelectric potential sensors 31-1 to 31-7 among the myoelectric potential data corresponding to myoelectric potential sensors 31-1 to 31-8 are above the threshold, the force state is identified as one in which the hand is spread out on the desk and the thumb is tensed.

Alternatively, as shown in FIG. 5, the force state identification unit 52 can identify the force state of the user's fingers using a recognition model generated by machine learning (e.g., k-nearest neighbor method, support vector machine, neural network, etc.) using as input EMG data for a relaxed state and a state in which each of the five fingers is tense. That is, the force state identification unit 52 can input EMG data supplied from the communication unit 41 to the recognition model and output a recognition result indicating the force state of the user's fingers recognized from the EMG data. Note that when the force state identification unit 52 identifies the force state of the user's fingers using a recognition model, it may be combined with the recognition model used by the hand shape identification unit 51 to form a single recognition model.

The initial movement detection unit 53 detects that the user has performed an initial movement based on the user's hand shape identified by the hand shape identification unit 51 and the force state of the user's fingers identified by the force state identification unit 52. For example, the user performs an initial movement when they want to start operating the information processing terminal 13 by inputting with strained fingers.

Here, the initial action is a motion pattern formed by a gesture of applying pressure to the fingers in a combination that is not normally performed when the user grasps the real object 21, and a motion pattern that can be detected based on the myoelectric potential measured by the myoelectric potential measuring device 12 is used. The motion pattern of the initial action is registered in advance in the registration data of the memory unit 42, and the initial action can be registered in any motion pattern. For example, the registration data may be registered so that the initial action differs depending on the real object 21 being grasped. Note that any graspable object in the real world can be used as the real object 21; for example, the information processing terminal 13 or a part of the user's own body may be used as the real object 21.

Figure 6 shows an example of an initial motion, with the numbers shown indicating the order in which the fingers are pressed. For example, as shown in Figure 6A, an initial motion can be used in which the index finger, middle finger, ring finger, little finger, and thumb are pressed in order with the fingers gripping a rod-shaped real object 21A. Alternatively, as shown in Figure 6B, an initial motion can be used in which the fingers are pinching a small rectangular parallelepiped real object 21B with the thumb, index finger, and middle finger, and the ring finger and little finger are pressed in order without applying force to real object 21B.

Other initial movements that can be used include simultaneously applying pressure to the index finger and little finger, applying pressure to the thumb three times in succession within a short period of time, applying pressure to the index finger and thumb for five seconds, applying pressure to each finger in turn every second, and applying pressure in two stages, starting with a weak pressure followed by a stronger pressure.

For example, the initial movement is performed with the primary purpose of triggering an operational input to the information processing terminal 13. This prevents operational input that goes against the user's intention and makes it possible to explicitly indicate to the information processing terminal 13 that an operational input has begun. Furthermore, in the terminal operation system 11, after the initial movement is detected, power consumption can be reduced by acquiring myoelectric potential data for identifying the user's hand shape and finger force state at a high sampling rate. For example, in the terminal operation system 11, it is preferable to adopt, as the initial movement form, a movement in which the fingers are tense for a relatively long period of time so that myoelectric potential data can be acquired even at a low sampling rate, so that myoelectric potential data can be acquired at a low sampling rate for detecting the initial movement.

Furthermore, the second purpose of the initial operation is to be used as a calibration (initialization process) for setting a threshold value that the operation execution unit 55 uses to detect the force state of the fingers. For example, the maximum values of the myoelectric potential data obtained during the initial operation when each finger is exerted force can be stored, and an arbitrary percentage of these maximum values can be set as the threshold value that the operation execution unit 55 uses.

For example, the left side of Figure 7 shows an example of myoelectric potential data obtained from the myoelectric potential signals output from myoelectric potential sensors 31-1 to 31-8 when the fingers are tense for the first time in the initial movement, as shown by the dashed-dotted line, and is recorded as myoelectric potential data indicating the finger shape in the initial movement. The right side of Figure 7 shows an example of myoelectric potential data obtained from the myoelectric potential signals output from myoelectric potential sensors 31-1 to 31-8 when the fingers are tense for the first time in the initial movement. In this way, the maximum value of the myoelectric potential data when the fingers are tense for the first time is stored, and an arbitrary ratio of this maximum value, as shown by the dashed-dotted line, can be set as the threshold value used by the operation execution unit 55 to detect the force state when the fingers are tense.

The terminal operation system 11 may identify the real object 21 being held or supported by the user from the motion pattern of the initial motion. Different initial motions may then be associated with different real objects 21 that are held or supported with similar motion patterns. The terminal operation system 11 may also determine the frame rate for measuring the myoelectric potential from the results of the initial motion; for example, the frame rate may be set to be lowered if the myoelectric potential fluctuates significantly and is easy to identify.

If the user's hand shape identified by the hand shape identification unit 51 is registered in the registration data stored in the memory unit 42, the operation mode selection unit 54 selects the operation mode associated with that hand shape. Furthermore, if the user's hand shape identified by the hand shape identification unit 51 is not registered in the registration data stored in the memory unit 42, the operation mode selection unit 54 selects the standard mode as the operation mode. Note that, in addition to having the operation mode selection unit 54 select an operation mode, the user may also directly operate the information processing terminal 13 to select a desired operation mode. Furthermore, the operation mode associated with the user's hand shape and registered in the registration data in the memory unit 42 can be set arbitrarily, and the operation content for each operation mode can also be set arbitrarily.

In addition, in the terminal operation system 11, if it is inferred from the myoelectric potential data that the user's hand shape matches a pre-registered hand shape at the time of the initial movement, even if the user does not perform an initial movement, the operation mode selection unit 54 can select the operation mode corresponding to that hand shape.

If the operation execution unit 55 detects that the force state of the user's fingers identified by the force state identification unit 52 after detecting the initial operation corresponds to the force state of the operation content in the operation mode selected by the operation mode selection unit 54, it executes processing in accordance with the operation content corresponding to that force state. For example, the operation execution unit 55 can detect the force state of the user's fingers based on a binary value (0, 1) indicating whether the strain of each finger exceeds a threshold. Alternatively, the operation execution unit 55 may detect the force state of the user's fingers based on a linear output (0 to 100%) of the strain of each finger.

With reference to Figures 8 and 9, we will explain the hand shape, operation mode, and operation content.

As shown in FIG. 8, when the user's finger position is such that the right hand is open and all fingertips are in contact with an object, the operation mode selection unit 54 selects the mouse operation mode as the operation mode. In the mouse operation mode, the operation execution unit 55 executes processing according to the operation content corresponding to a left button click depending on the force state of the index finger, and executes processing according to the operation content corresponding to a double click depending on the force state of the index finger. Similarly, in the mouse operation mode, the operation execution unit 55 executes processing according to the operation content corresponding to a right button click depending on the force state of the middle finger, executes processing according to the operation content corresponding to a back button click depending on the force state of the thumb, and executes processing according to the operation content corresponding to a forward button click depending on the force state of the ring finger.

When the user's fingering position is such that the left hand is open and all fingertips other than the little finger are in contact with an object, the operation mode selection unit 54 selects the cross key operation mode as the operation mode. In the cross key operation mode, the operation execution unit 55 executes processing according to the operation content corresponding to the upward direction depending on the force state of the middle finger, and executes processing according to the operation content corresponding to the left direction depending on the force state of the ring finger. Similarly, in the cross key operation mode, the operation execution unit 55 executes processing according to the operation content corresponding to the right direction depending on the force state of the index finger, and executes processing according to the operation content corresponding to the downward direction depending on the force state of the thumb.

When the user's finger shape is gripping the steering wheel, the operation mode selection unit 54 selects the map operation mode as the operation mode. In the map operation mode, the operation execution unit 55 executes processing according to the operation content to enlarge the map in accordance with the force state changing from tension in the index finger to tension in the middle finger, and executes processing according to the operation content to reduce the map in accordance with the force state changing from tension in the middle finger to tension in the index finger. Similarly, in the map operation mode, the operation execution unit 55 executes processing according to the operation content to play the next guidance in accordance with the force state of continuous tension in the thumb.

When the user's finger position is such that the earphones are pinched with the index finger and thumb, the operation mode selection unit 54 selects the volume operation mode as the operation mode. In the volume operation mode, the operation execution unit 55 executes processing according to the operation content to increase the volume depending on the force state of the index finger, and executes processing according to the operation content to decrease the volume depending on the force state of the thumb. Similarly, in the volume operation mode, the operation execution unit 55 executes processing according to the operation content to stop or play depending on the force state of all fingers.

As shown in A of Figure 9, if the user's finger shape is not registered in the registration data stored in the memory unit 42, the operation mode selection unit 54 selects the standard mode as the operation mode. In the standard mode, the operation execution unit 55 executes processing according to the operation content corresponding to moving forward depending on the force state of the index finger, executes processing according to the operation content corresponding to moving back depending on the force state of the middle finger, and executes processing according to the operation content corresponding to making a decision depending on the force state of the thumb.

When the user's finger shape is the OK sign (a hand sign in which the thumb and index finger form a circle) as shown in B of Figure 9, the operation mode selection unit 54 selects the Yes-No mode as the operation mode. In the Yes-No mode, the operation execution unit 55 executes processing according to the operation content corresponding to Yes depending on the force state of the index finger and thumb, and executes processing according to the operation content corresponding to No depending on the force state of the middle finger and thumb.

Note that the hand shapes, operation modes, and operation contents described with reference to Figures 8 and 9 are merely examples, and other hand shapes, operation modes, and operation contents may also be used.

As described above, the terminal operation system 11 can perform an initial operation using the strength of the user's five fingers while the user is holding or supporting any real object 21, thereby enabling the user to begin inputting operations into the information processing terminal 13. Based on the user's hand shape and finger force state, the terminal operation system 11 can then execute processing in accordance with the corresponding operation content, for example, different processing depending on the force exerted by each of the user's five fingers.

Furthermore, the terminal operation system 11 can use any graspable real object 21 in the real world to input operations to the information processing terminal 13. For example, even if different real objects 21 are used, the terminal operation system 11 can execute processing according to the same operation content as long as the hand shape and combination of fingers used to apply pressure are the same. In other words, the terminal operation system 11 can operate the information processing terminal 13 without being limited to using known real objects 21 such as pen-shaped, elongated objects. In this way, the terminal operation system 11 can use a variety of arbitrary real objects 21 to achieve greater operability with a higher degree of freedom.

<Processing example of electromyography measurement processing and operation execution processing>
The myoelectric potential measurement process and operation execution process executed in the terminal operation system 11 will be described with reference to the flowcharts shown in FIGS.

The myoelectric potential measurement process executed by the myoelectric potential measuring device 12 will be described with reference to the flowchart shown in Figure 10.

In step S11, the communication unit 34 determines whether or not it has received a signal instructing the start of communication transmitted from the communication unit 41 of the information processing terminal 13, and waits until it determines that it has received a signal instructing the start of communication. If the communication unit 34 determines that it has received a signal instructing the start of communication, the process proceeds to step S12.

In step S12, the signal acquisition unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 and supplies them to the signal processing unit 33.

In step S13, the signal processing unit 33 acquires the myoelectric potential data obtained as a result of performing signal processing such as noise removal and Fourier transform processing on the myoelectric potential signal supplied in step S12, and supplies this data to the communication unit 34.

In step S14, the communication unit 34 transmits the myoelectric potential data provided in step S13 to the information processing terminal 13.

In step S15, the communication unit 34 determines whether or not a signal instructing the end of communication transmitted from the communication unit 41 of the information processing terminal 13 has been received. If the communication unit 34 determines that a signal instructing the end of communication has not been received, the process returns to step S12, and the same process is repeated thereafter. On the other hand, if the communication unit 34 determines that a signal instructing the end of communication has been received, the process proceeds to step S16.

In step S16, various processes are performed to terminate the operation of the EMG measuring device 12, and then the EMG measurement process is terminated.

The operation execution process executed on the information processing terminal 13 will be described with reference to the flowchart shown in Figure 11.

In step S21, the communication unit 41 transmits a signal to the communication unit 34 of the electromyography measuring device 12 instructing it to start communication.

In step S22, the communication unit 41 receives the myoelectric potential data transmitted from the communication unit 34 in step S14 of FIG. 10 and supplies it to the hand shape identification unit 51 and the force state identification unit 52.

In step S23, the hand posture identification unit 51 identifies the user's hand posture based on the myoelectric potential data provided in step S22, and provides the recognition result indicating the user's hand posture to the initial action detection unit 53 and the operation mode selection unit 54.

In step S24, the force state identification unit 52 identifies the force state of the user's finger based on the EMG data supplied in step S22, and supplies the recognition result indicating the force state of the user's finger to the initial action detection unit 53 and the operation execution unit 55.

In step S25, the initial movement detection unit 53 determines whether an initial movement by the user has been detected based on the user's hand shape identified in step S23 and the force state of the user's fingers identified in step S24.

If the initial action detection unit 53 determines in step S25 that an initial action by the user has been detected, processing proceeds to step S26.

In step S26, the operation mode selection unit 54 determines whether the user's hand shape identified in step S23 is registered in the registration data stored in the memory unit 42.

If the operation mode selection unit 54 determines in step S26 that the user's hand shape is registered in the registration data, the process proceeds to step S27. In step S27, the operation mode selection unit 54 selects the operation mode associated in the registration data with the user's hand shape identified in step S23, and the process proceeds to step S30.

On the other hand, if the operation mode selection unit 54 determines in step S26 that the user's hand shape is not registered in the registration data (is unregistered), the process proceeds to step S28. In step S28, the operation mode selection unit 54 selects the standard mode as the operation mode, and the process proceeds to step S30.

On the other hand, if the initial action detection unit 53 determines in step S25 that an initial action by the user has not been detected, processing proceeds to step S29.

In step S29, the operation mode selection unit 54 determines whether the user's hand shape identified in step S23 matches the hand shape at the time of initial operation. If the operation mode selection unit 54 determines that the user's hand shape does not match the hand shape at the time of initial operation, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the operation mode selection unit 54 determines that the user's hand shape matches the hand shape at the time of initial operation, an operation mode corresponding to that hand shape is selected, and the process proceeds to step S30.

In step S30, the operation execution unit 55 determines whether the force state of the user's finger identified by the force state identification unit 52 after the initial operation is detected corresponds to the force state of the operation content in the currently selected operation mode.

If the operation execution unit 55 determines in step S30 that a force state corresponding to the force state of the operation content in the selected operation mode has not been detected, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the operation execution unit 55 determines in step S30 that a force state corresponding to the force state of the operation content in the selected operation mode has been detected, the process proceeds to step S31.

In step S31, the operation execution unit 55 executes processing according to the operation content corresponding to the force state of the user's finger detected in step S30.

In step S32, the calculation unit 43 determines whether to end the operation execution process.

If the calculation unit 43 determines in step S32 not to end the operation execution process, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the calculation unit 43 determines to end the operation execution process, the process proceeds to step S33.

In step S33, the communication unit 41 sends a signal to the communication unit 34 of the electromyography measuring device 12 instructing it to end communication, and then the operation execution process ends.

As described above, the terminal operation system 11 can utilize a variety of arbitrary real objects 21 to achieve greater operability.

<Examples of use and applications of the terminal operation system>
12 to 15, examples of use and applications of the terminal operation system 11 will be described.

Figure 12 is a diagram illustrating an example of a usage example in which a smartwatch is used as the terminal operation system 11.

For example, when a smartwatch is used as the terminal operation system 11, in the usage example shown in FIG. 12, a myoelectric potential measuring device 12 is incorporated into the band of the smartwatch, and an information processing terminal 13 provided on the main body of the smartwatch performs the operation execution process described above. Also, in the usage example shown in FIG. 12, the handlebars of a bicycle are used as the real object 21C.

If the user's fingers are gripping the steering wheel, the map operation mode is selected as the operation mode, as shown in A of Figure 12.

When a user activates a map app on the smartwatch, which is the terminal operation system 11, to travel to a destination by bicycle, a map is displayed on the display unit of the terminal operation system 11, and navigation to the destination begins, as shown in Figure 12B. Then, the user performs an initial motion, for example, by simultaneously pressing their index finger and little finger, and then simultaneously pressing their middle finger and ring finger, thereby initiating operation input to the terminal operation system 11 and starting to pedal the bicycle.

For example, if the user wants to know when to make the next right turn, by continuously applying pressure to the thumb, the display of the terminal operation system 11 will display a navigation message saying "Turn right in 450m" in accordance with the operation content corresponding to the pressure applied, as shown in Figure 12C.

Furthermore, when a user wants to check a destination on a map but the destination cannot be displayed at the current map scale, by first pressing down on the index finger and then pressing down on the middle finger, the map displayed on the display unit of the terminal operation system 11 will be enlarged according to the operation content corresponding to the state of pressure, as shown in D of Figure 12. Note that the map can be enlarged so that the magnification changes linearly depending on the strength of the pressure applied to the fingers at this time.

FIG. 13 is a diagram illustrating an example of a user cooking using AR glasses as the information processing terminal 13 and a spatula as the real object 21D.

For example, a user wears AR glasses, which are information processing terminal 13, on their head and plays a recipe video, and is cooking while holding a spatula in their right hand, which is wearing myoelectric potential measuring device 12, and a frying pan in their left hand. At this time, there may be a situation where the user wants to return to the recipe video displaying a list of ingredients to check the next ingredient to be added, but cannot take their hands off the food to avoid burning it. In such a situation, the user performs an initial movement by pressing the index finger, middle finger, and thumb in the right hand holding the spatula, in that order.

At this time, if the hand shape holding the spatula is not registered in the registration data of the memory unit 42, the standard mode (A in FIG. 13) is selected as the operation mode. Then, on the AR glasses that are the information processing terminal 13, operation guides as shown by dashed lines are displayed superimposed on each finger according to the operation content of the standard mode, as shown in B in FIG. 13. That is, an operation guide representing the operation content "forward" is displayed superimposed on the index finger, an operation guide representing the operation content "back" is displayed superimposed on the middle finger, and an operation guide representing the operation content "play" is displayed superimposed on the thumb.

Therefore, when the user applies pressure to the middle finger of their right hand while holding a spatula, the playback position of the recipe video is rewound by 30 seconds in increments according to the operation content that corresponds to the pressure applied, and the user must repeatedly apply pressure to the middle finger until the playback position of the recipe video returns to the scene they wish to view. Then, when the user applies pressure to their thumb at the moment the playback position of the recipe video returns to the scene they wish to view, the recipe video is played according to the operation content that corresponds to the pressure applied.

In addition, in the terminal operation system 11, when the user has successfully executed an operation by applying pressure to the finger, the AR glasses, which are the information processing terminal 13, may notify the user by video, audio, or vibration. For example, as shown in C of FIG. 13, when the user has successfully executed the operation content "back" by applying pressure to the middle finger, the display of the operation guide may be slightly changed to notify the user. Furthermore, for example, the notification may be given in a different color for each finger, or a different musical scale for each finger.

Furthermore, as time passes from the start of operation of the terminal operation system 11, it is expected that the user will become fatigued and the strength of the force applied to their fingers will decrease, so the threshold for the strength of the force required for operation may be lowered as time passes from the start of operation.

For example, as shown in A of FIG. 14, if a certain amount of time has not passed since the start of the operation, the operation guide superimposed on the user's fingers gripping the rod-shaped real object 21E is displayed in normal size. Then, as shown in B of FIG. 14, if a certain amount of time has passed since the start of the operation, the operation guide superimposed on the user's fingers gripping the rod-shaped real object 21E is displayed smaller than normal size to indicate that the threshold force required for the operation has decreased. In this case, conversely, the operation guide may be displayed larger than normal size to encourage the user to exert more force when operating.

This section explains an application example in which a camera is mounted on the information processing terminal 13 and a real object 21 can be detected using the camera.

For example, when the information processing terminal 13 identifies a known real object 21 with a camera, it can select an operation mode corresponding to that known real object 21 without performing any initial operation. On the other hand, when the information processing terminal 13 identifies an unknown real object 21 with a camera, it can select the standard mode without performing any initial operation. Furthermore, when the information processing terminal 13 is capable of superimposing display using AR, it may display a hand icon superimposed on the real object 21 identified with the camera, and may also display the operation details of operations that can be performed using that real object 21. Furthermore, when the information processing terminal 13 is capable of detecting the user's hand with a camera, it can recognize the hand and finger shape not only using myoelectric potential data but also using an image of the hand captured by the camera.

In addition, the information processing terminal 13 may use a camera to identify the relative position of the user's finger as it is being pressed, and change the processing according to that position. For example, as shown in FIG. 15, when an operation is performed on a keyboard, when the index finger is pressed at a certain position, it is assumed that an operation to input the letter "J" has been performed, and when the index finger is pressed at a position approximately 2 cm to the right from that position, it is assumed that an operation to input the letter "K" has been performed. In this way, even if the index finger is pressed with the same force, the processing performed can be changed according to the position.

Furthermore, as an application example, the terminal operation system 11 may select a specific operation mode and accept operations for the information processing terminal 13 based on the myoelectric potential data for a certain period of time immediately after some action occurs on the information processing terminal 13 side, even if no initial operation is performed assuming that an operation input will be made. Then, when operation acceptance has begun in the specific operation mode, the sampling frequency for measuring the myoelectric potential data may be increased.

Furthermore, in the terminal operation system 11, if the information processing terminal 13 is a smartphone, when there is a voice message (such as an incoming phone call), the terminal may be configured to accept operations on the information processing terminal 13 based on the electromyogram data for a certain period of time thereafter. At this time, it is expected that, for example, an operation to answer or reject the call will be performed based on the electromyogram data, so a Yes-No mode (see B in Figure 9) can be selected.

Furthermore, in the terminal operation system 11, in preparation for the occurrence of an operation unintended by the user, an operation to cancel the most recent operation input (cancel operation) can be set to be available in all operation modes. For example, the cancel operation can be a force state in which all fingers are spread apart, or a force state in which all fingers are tensed. Furthermore, if the cancel operation is performed a certain number of times or more, the terminal operation system 11 may consider the operation input to be canceled to be a false detection and may prevent the processing corresponding to that operation input from being executed, and may learn from the results of the cancel operation and correct the operation mode.

Furthermore, the terminal operation system 11 can be used for sports analysis. For example, when gripping a golf club (real object 21), the terminal operation system 11 can save the way each finger applies force rather than explicit operation input, analyze it as a golfing motion, and provide feedback such as "Relax your thumb." Similarly, the terminal operation system 11 can measure the user's level of concentration on driving from the way the steering wheel (real object 21) is held while driving.

Furthermore, the terminal operation system 11 can identify individuals based on the posture in which the real object 21 is held, the force used when holding the real object 21, and so on. For example, processing may be performed by referencing registered data for each individual.

In addition, if the terminal operation system 11 can identify the user's finger shape and finger force state, it may use measurement data obtained by using sensors to detect changes in muscle sounds, ultrasound, electromagnetic waves, capacitance, etc., in addition to using myoelectric potential data.

Furthermore, the terminal operation system 11 may be configured so that the memory unit 42 and some blocks of the calculation unit 43 are provided on a remote server, and the server and the information processing terminal 13 are constantly connected via a network. In this configuration, the information processing terminal 13 needs to be equipped with a function for synchronizing with the remote server via wired or wireless communication. Also, in this configuration, the operation form of the initial operation, the correspondence between the user's finger shapes and operation modes, the operation content for each operation mode, etc. are stored on the remote server, so even if a different information processing terminal 13 is used, the same operation input can be performed as long as the same real object 21 is being used.

<Example of computer configuration>
Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

FIG. 16 is a block diagram showing an example configuration of one embodiment of a computer on which a program that executes the series of processes described above is installed.

In a computer, a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, and EEPROM (Electronically Erasable and Programmable Read Only Memory) 104 are interconnected by a bus 105. An input/output interface 106 is also connected to the bus 105, and the input/output interface 106 is connected to the outside.

In a computer configured as described above, the CPU 101 loads programs stored in, for example, ROM 102 and EEPROM 104 into RAM 103 via bus 105 and executes them, thereby performing the above-mentioned series of processes. Furthermore, programs executed by the computer (CPU 101) can be written in advance to ROM 102, or can be installed or updated in EEPROM 104 from the outside via input/output interface 106.

In this specification, the processing performed by a computer in accordance with a program does not necessarily have to be performed chronologically in the order described in the flowchart. In other words, the processing performed by a computer in accordance with a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

Furthermore, the program may be processed by a single computer (processor), or may be distributed across multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

Furthermore, for example, a configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, configurations other than those described above may of course be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

Furthermore, for example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

Furthermore, for example, the above-mentioned program can be executed on any device. In that case, it is sufficient if the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

Furthermore, for example, each step described in the flowchart above can be executed by a single device, or can be shared and executed by multiple devices. Furthermore, if a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device, or can be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can also be executed collectively as a single step.

In addition, the processing of the steps that describe a program executed by a computer may be executed chronologically in the order described in this specification, or may be executed in parallel, or individually at the required timing, such as when a call is made. In other words, as long as no contradictions arise, the processing of each step may be executed in an order different from that described above. Furthermore, the processing of the steps that describe this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

Note that the multiple present technologies described in this specification can be implemented independently and individually, provided no contradictions arise. Naturally, any multiple present technologies can also be implemented in combination. For example, some or all of the present technologies described in any embodiment can be implemented in combination with some or all of the present technologies described in other embodiments. Furthermore, some or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

<Configuration combination example>
The present technology can also be configured as follows.
(1)
an acquisition unit that acquires measurement data corresponding to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and a calculation unit that estimates the user's hand shape and hand force state based on the measurement data and executes processing according to the user's operation.
(2)
The calculation unit
a hand and finger posture specifying unit that specifies a hand and finger posture representing the shape of the user's hand and fingers based on the measurement data;
and a force state specifying unit that specifies, based on the measurement data, which of the user's five fingers is straining and a force state that indicates the strength of the strain on that finger.
(3)
The information processing device described in (2) above, wherein the calculation unit further includes an initial movement detection unit that detects that the user has performed an initial movement based on the user's hand shape identified by the hand shape identification unit and the force state of the user's fingers identified by the force state identification unit.
(4)
The information processing device according to (3) above, wherein the initial action is a gesture in which the user, who is holding or supporting the given real object, applies pressure to the fingers in a predetermined combination.
(5)
The information processing device described in (3) above, wherein the calculation unit further includes an operation mode selection unit that, when the initial movement detection unit detects that the user has performed an initial movement, selects an operation mode associated with the user's hand shape identified by the hand shape identification unit.
(6)
The information processing device described in (5) above, wherein the calculation unit further includes an operation execution unit that, when detecting that the force state of the user's finger identified by the force state identification unit is a force state corresponding to the force state of the operation content in the operation mode selected by the operation mode selection unit, executes processing in accordance with the operation content corresponding to the force state.
(7)
The information processing device according to (6) above, wherein the operation mode selection unit selects a map operation mode as the operation mode when the hand posture identification unit identifies that the user's hand posture is gripping a steering wheel.
(8)
When the map operation mode is selected by the operation mode selection unit, the operation execution unit
The system executes the process according to the operation content corresponding to "zooming in on the map" according to the strength state of the index finger, which changes from strength of the middle finger to strength of the index finger.
As the force applied to the finger changes from the middle finger to the index finger, the system executes the operation corresponding to "zooming out the map."
The information processing device according to (7) above, which executes processing according to operation content corresponding to "play next guidance" depending on the force state of the thumb.
(9)
The information processing device according to (6), wherein the operation mode selection unit selects a standard mode as the operation mode when the user's hand posture identified by the hand posture identification unit is unregistered.
(10)
When the standard mode is selected by the operation mode selection unit, the operation execution unit:
Depending on the force applied to the index finger, the system executes the operation corresponding to "Go forward."
Depending on the force applied to the middle finger, the system executes the operation corresponding to "back";
The information processing device according to (9) above, wherein a process is executed according to an operation content corresponding to "decide" depending on a force state of the thumb.
(11)
The information processing device according to any one of (5) to (10) above, wherein, even if the user does not perform an initial operation, when the user's hand shape identified by the hand shape identification unit matches the hand shape at the time of the initial operation, the operation mode selection unit selects an operation mode corresponding to the hand shape.
(12)
The information processing device according to any one of (1) to (11) above, wherein the measurement data is myoelectric potential data obtained as a result of measuring myoelectric potential of the user's arm.
(13)
the information processing device is a smart watch having a band incorporating a myoelectricity measuring device that outputs the myoelectricity data,
The information processing device according to (12) above, wherein at least the calculation unit is provided in a main body of the smartwatch.
(14)
The information processing device
Acquiring measurement data according to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and estimating the user's hand shape and hand force state based on the measurement data, and executing processing according to the user's operation.
(15)
The computer of the information processing device
Acquiring measurement data according to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and estimating the user's hand shape and hand force state based on the measurement data, and executing processing according to the user's operation.

Note that this embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be present.

11. Terminal operation system, 12. EMG measurement device, 13. Information processing terminal, 21. Real object, 31. EMG sensor, 32. Signal acquisition unit, 33. Signal processing unit, 34. Communication unit, 41. Communication unit, 42. Memory unit, 43. Calculation unit, 51. Hand shape determination unit, 52. Force state determination unit, 53. Initial movement detection unit, 54. Operation mode selection unit, 55. Operation execution unit

Claims (15)

an acquisition unit that acquires measurement data corresponding to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and a calculation unit that estimates the user's hand shape and hand force state based on the measurement data and executes processing according to the user's operation. The calculation unit
a hand and finger posture specifying unit that specifies a hand and finger posture representing the shape of the user's hand and fingers based on the measurement data;
The information processing device according to claim 1 , further comprising: a force state specifying unit that specifies, based on the measurement data, which of the user's five fingers is straining and a force state that indicates the strength of the strain on that finger. 3. The information processing device according to claim 2, wherein the calculation unit further includes an initial movement detection unit that detects that the user has performed an initial movement, based on the user's hand shape identified by the hand shape identification unit and the force state of the user's fingers identified by the force state identification unit. The information processing device according to claim 3 , wherein the initial motion is a motion form based on a gesture in which the user, who is holding or supporting the given real object, applies pressure to the fingers in a predetermined combination. 4. The information processing device according to claim 3, wherein the calculation unit further comprises an operation mode selection unit that selects an operation mode associated with the user's hand posture identified by the hand posture identification unit when the initial movement detection unit detects that the user has performed an initial movement. 6. The information processing device according to claim 5, wherein the calculation unit further includes an operation execution unit that, when detecting that the force state of the user's finger identified by the force state identification unit corresponds to a force state of operation content in the operation mode selected by the operation mode selection unit, executes processing in accordance with the operation content corresponding to the force state. The information processing device according to claim 6 , wherein the operation mode selection unit selects a map operation mode as the operation mode when the hand posture identification unit identifies that the user's hand posture is gripping a steering wheel. When the map operation mode is selected by the operation mode selection unit, the operation execution unit
The system executes the process according to the operation content corresponding to "zooming in on the map" according to the strength state of the index finger, which changes from strength of the middle finger to strength of the index finger.
As the force applied to the finger changes from the middle finger to the index finger, the system executes the operation corresponding to "zooming out the map."
The information processing device according to claim 7 , wherein the device executes a process according to an operation content corresponding to “playback of next guidance” depending on the force state of the thumb. The information processing apparatus according to claim 6 , wherein the operation mode selection unit selects a standard mode as the operation mode when the user's hand posture identified by the hand posture identification unit is unregistered. When the standard mode is selected by the operation mode selection unit, the operation execution unit:
Depending on the force applied to the index finger, the system executes the operation corresponding to "Go forward."
Depending on the force applied to the middle finger, the system executes the operation corresponding to "back";
The information processing device according to claim 9 , wherein the device executes a process according to an operation content corresponding to “decide” depending on the force state of the thumb. 6. The information processing device according to claim 5, wherein, even if the user does not perform an initial operation, when the user's hand shape identified by the hand shape identification unit matches the hand shape at the time of the initial operation, the operation mode selection unit selects the operation mode corresponding to the hand shape. The information processing device according to claim 1 , wherein the measurement data is myoelectric potential data obtained as a result of measuring myoelectric potential of the user's arm. the information processing device is a smart watch having a band incorporating a myoelectricity measuring device that outputs the myoelectricity data,
The information processing device according to claim 12 , wherein at least the calculation unit is provided in a main body of the smartwatch. The information processing device
Acquiring measurement data according to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and estimating the user's hand shape and hand force state based on the measurement data, and executing processing according to the user's operation. The computer of the information processing device
Acquiring measurement data according to the movement of the user's hand and fingers while grasping or supporting an arbitrary real object;
and estimating the user's hand shape and hand force state based on the measurement data, and executing processing according to the user's operation.