CN116166161A - Interaction method based on multi-level menu and related equipment - Google Patents

Abstract

The application provides an interaction method based on a multi-level menu and related equipment, wherein the type of a first operation object is determined by identifying a first gesture of the first operation object aiming at a plurality of operation objects of the multi-level menu, when the first operation object is the identification of a first submenu, the first submenu is displayed, and when the first operation object is the identification of a first operation, the first operation is executed. The mode of completing interaction through gesture recognition can reduce the false touch rate of the existing interaction mode to a certain extent.

Description

Interaction method and related equipment based on multi-level menu

technical field

The present application relates to the technical field of human-computer interaction, in particular to an interaction method based on a multi-level menu and related equipment.

Background technique

With the development of intelligent software and hardware technology, there has been an extended reality (Extented Reality, XR) technology realized through computer technology and wearable devices. XR technology may further include virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR) and mixed reality (Mixed Reality, MR). VR is a technology that uses computer simulation to generate a virtual environment. AR is a technology that integrates virtual information with the real world. MR is a hybrid of both AR and VR.

During actual use, users can interact with virtual objects presented in VR/AR/MR scenes. For example, when a multi-level menu is presented in the scene, the user makes a selection by clicking a triggerable object in the multi-level menu. However, there is a high rate of false touches in this way of interacting with multi-level menus.

Contents of the invention

In view of this, the purpose of this application is to propose an interaction method and related equipment based on multi-level menus, so as to solve or partially solve the aforementioned problems.

In the first aspect of the present application, an interaction method based on a multi-level menu is provided, the multi-level menu includes a plurality of operation objects, and the method includes:

determining a type of the first operating object in response to recognizing a first gesture directed to a first operating object of the plurality of operating objects;

displaying the first submenu in response to determining that the first operation object is an identifier of the first submenu; or

In response to determining that the first operation object is an identifier of a first operation, the first operation is executed.

In the second aspect of the present application, an interaction device based on a multi-level menu is provided, including:

A recognition module configured to determine the type of the first operation object in response to recognizing a first gesture directed to a first operation object of the plurality of operation objects;

an execution module configured to display the first submenu in response to determining that the first operation object is a first submenu; or

In response to determining that the first operation object is an identifier of a first operation, the first operation is executed.

In the third aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the program, the computer program described in the first aspect is implemented. Methods.

In a fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method as described in the first aspect .

A fifth aspect of the present application provides a computer program product. When the computer program is executed by a processor, the method described in the first aspect is implemented.

It can be seen from the above that the multi-level menu-based interaction method and related equipment provided by the present application determine the first The type of the operation object. When the first operation object is the identifier of the first submenu, the first submenu is displayed; when the first operation object is the identifier of the first operation, the first operation is executed. In this application, gesture recognition is used to complete the interaction, which can reduce the false touch rate of existing interaction methods to a certain extent.

Description of drawings

In order to more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are only for this application Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1 shows a schematic diagram of an exemplary extended reality system 100 according to an embodiment of the present application.

FIG. 2 shows a schematic diagram of an exemplary multi-level menu-based interaction screen 200 .

FIG. 3A shows a schematic diagram of an exemplary screen 300 according to an embodiment of the present application.

FIG. 3B shows a schematic diagram of yet another exemplary screen 300 according to an embodiment of the present application.

Fig. 3C shows a schematic diagram of an exemplary gesture according to an embodiment of the present application.

FIG. 3D shows a schematic diagram of yet another exemplary screen 300 according to an embodiment of the present application.

FIG. 3E shows a schematic diagram of another exemplary screen 300 according to an embodiment of the present application.

Fig. 3F shows a schematic diagram of another exemplary gesture according to an embodiment of the present application.

FIG. 3G shows a schematic diagram of yet another exemplary screen 300 according to an embodiment of the present application.

FIG. 3H shows a schematic diagram of yet another exemplary screen 300 according to an embodiment of the present application.

FIG. 3I shows a schematic diagram of yet another exemplary screen 300 according to an embodiment of the present application.

FIG. 4A shows a schematic diagram of an exemplary multi-level menu-based interaction screen 400 .

FIG. 4B shows a schematic diagram of another exemplary multi-level menu-based interaction screen 400 .

FIG. 5A shows a schematic diagram of an exemplary screen 500 according to an embodiment of the present application.

FIG. 5B shows a schematic diagram of another exemplary screen 500 according to an embodiment of the present application.

FIG. 5C shows a schematic diagram of an exemplary screen 510 according to an embodiment of the present application.

FIG. 5D shows a schematic diagram of another exemplary screen 510 according to an embodiment of the present application.

FIG. 5E shows a schematic diagram of an exemplary screen according to an embodiment of the present application.

FIG. 6A shows a schematic diagram of an exemplary screen 600 according to an embodiment of the present application.

FIG. 6B shows a schematic diagram of another exemplary screen 600 according to an embodiment of the present application.

FIG. 6C shows a schematic diagram of yet another exemplary screen 600 according to an embodiment of the present application.

FIG. 6D shows a schematic diagram of yet another exemplary screen 600 according to an embodiment of the present application.

FIG. 7 shows a flowchart of an exemplary multi-level menu-based interaction method 700 according to an embodiment of the present application.

Fig. 8 shows a schematic diagram of an exemplary multi-level menu-based interaction device according to an embodiment of the present application.

FIG. 9 is a schematic diagram of a more specific hardware structure of an electronic device provided by this embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application shall have the usual meanings understood by those skilled in the art to which the present application belongs. "First", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

As mentioned above, users can interact with virtual objects presented in VR/AR/MR scenes. For example, when a multi-level menu is presented in the scene, the user makes a selection by clicking a triggerable object in the multi-level menu. However, there are multiple operation objects in the multi-level menu, and multiple operation objects are displayed in the multi-level menu in a side-by-side and/or parallel manner. Other triggerable objects adjacent to the targeted triggerable object, so this way of interacting with multi-level menus creates a false touch rate.

Fig. 1 shows a schematic diagram of an exemplary extended reality system 100 according to an embodiment of the present application.

As shown in FIG. 1 , the system 100 may include a head-mounted wearable device (for example, VR glasses) 104 , wearing gloves 106 , and an operating handle 108 . In some scenarios, a camera/camera 110 for taking pictures of the operator (user) 102 may also be provided. In some embodiments, when the aforementioned devices do not have processing functions, the system 100 may further include an external control device 112 for providing processing functions. The control device 112, for example, may be a computer device such as a mobile phone or a computer. In some embodiments, when any one of the foregoing devices is used as a control device or a main control device, it can realize information interaction with other devices in the system 100 through wired or wireless communication.

In some embodiments, as shown in FIG. 1 , the system 100 can also communicate with the server 114 and can obtain data from the server 114 , such as pictures, audio, video, and the like. In some embodiments, as shown in FIG. 1 , the server 114 can retrieve required data, such as pictures, audio, video, etc., from the database server 116 for storing data.

In the system 100 , the operator 102 can interact with the extended reality system 100 by using a head-mounted wearable device (for example, VR glasses) 104 , wearing gloves 106 , and operating a handle 108 . In some embodiments, the head-mounted wearable device 104, the wearable glove 106, and the operating handle 108 can all be provided with a collection unit for collecting information, and the system 100 can use the information collected by the collection unit to identify the posture of the operator 102, Gestures, etc., and then realize the interaction between the operator 102 and the extended reality system 100 based on the recognized gestures and gestures. There can be multiple types of acquisition units. For example, the head-mounted wearable device 104 can be provided with a camera or a charge-coupled device (CCD) image sensor for collecting human eye images, a speed sensor for collecting speed information or acceleration information of the head-mounted wearable device 104, an acceleration Sensors, angular velocity sensors (eg, gyroscopes), electrodes for collecting brainwave information, neuromuscular sensors for collecting information on neuromuscular responses, temperature sensors for collecting body surface temperature, and the like. For another example, the wearing glove 106 can also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (such as a gyroscope) for collecting speed information or acceleration information of the wearing glove 106, a neuromuscular sensor (neuromuscular sensor) for collecting neuromuscular response information. sensors), temperature sensors for collecting body surface temperature, etc. It should be noted that, in addition to being set on the head-mounted wearable device 104 and the glove 106, the aforementioned collection unit can also be set on the operating handle 108 in FIG. The attached method is set on the body part of the interactive operator 102, so as to collect relevant information of the body part, for example, velocity or acceleration or angular velocity information, or biological information collected by other sensors or acquisition units (for example, human eye images ( Contains pupil images), neuromuscular response information, brain wave information, body surface temperature, etc.). In some embodiments, the system 100 can also use the images captured by the camera/camera 110 to recognize the gestures of the operator 102's bare hands, and then complete the interaction with the operator 102 based on the recognized gestures.

The system 100 recognizes the gesture of the bare hand of the operator 102 based on the images captured by the capture unit or the camera/camera 110 . In some embodiments, the system 100 first performs skin color detection on the hand shape image captured by the acquisition unit or the camera/camera 110, and based on the result of the skin color detection, the hand shape image is segmented, so that the operator 102 in the hand shape image The target gesture is separated from the image background, and then features of the target gesture (such as the number of fingertips, finger joint points, feature vectors of the gesture, etc.) are extracted, and gesture recognition is performed based on the extracted features of the target gesture. For example, a gesture template may be used to identify the gesture by matching the characteristic parameters of the gesture to be recognized with the characteristic parameters of the gesture template. For another example, gestures may be recognized by classifying the extracted feature vectors based on statistical analysis techniques. For another example, a neural network learning model may be used to recognize gestures based on a large number of extracted gesture features.

An extended reality (XR) system 100 may allow an operator 102 to interact with the digital world within a simulated scene. The simulated scene may be a virtual scene or a scene combining virtual and real scenes. In some cases, it may be desirable for the operator 102 to interact with some virtual objects in the simulated scene. The virtual object may be a multi-level menu.

FIG. 2 shows a schematic diagram of an exemplary multi-level menu-based interaction screen 200 .

The screen 200 may be the screen viewed by the operator 102 through the head-mounted wearable device 104. As shown in FIG. A multi-level menu 202 is displayed on the screen 200 .

In some embodiments, multiple triggerable objects may be displayed in the multi-level menu 202, for example, as shown in FIG. submenu, or perform corresponding operations (for example, change to a different simulation scene), therefore, these operation objects 2022 can be considered as the triggerable objects.

After receiving the input request, in the related art, the operator 102 generally interacts with the multi-level menu 202 through a click operation.

As shown in FIG. 2 , in some embodiments, the multi-level menu 202 can be formed at a position closer to the operator 102 (which can be referred to as the near field), and can also be formed at a position farther from the operator 102 (which can be referred to as for the far field). Far Field From the perspective of visual effect, the multi-level menu 202 will be smaller than the screen 200 . In the related art, according to whether the multi-level menu 202 is in a far position (far field) or in a relatively close position (near field), and the different input media used by the operator 102 (for example, an operating handle or a hand), the Interaction with the multi-level menu 202 is realized, so that corresponding submenus are generated or corresponding operations are performed in the simulated scene.

As shown in FIG. 2 , in the related art, when the multi-level menu 202 is in a relatively close position (near field), the images (for example, the hand image 204) formed in the simulated scene by different input media used by the operator 102 ) front end forms pointing point 206. The hand image 204 is used to reflect the position of the hand of the operator 102 in the simulation scene, so that the operator 102 can operate based on the hand image 204 . The pointing point 206 formed on the front end of the hand image 204 is used as a point where the hand image 204 contacts a specific object in the multi-level menu 202, so that the operator 102 determines to trigger a certain available item in the multi-level menu 202 according to the position of the pointing point 206. The trigger object, which in turn triggers a triggerable object through a click action.

In addition, in some related technologies, if the operator uses the operating handle as an input medium, a pointing point will be formed at the front of the image formed by the operating handle in the simulated scene. Similar to the above-mentioned related technologies, the operator can determine to trigger a certain triggerable object in the multi-level menu according to the position of the pointing point, and then trigger a certain triggerable object through a click operation. In some other related technologies, if the multi-level menu is in a far position (far field), a ray pointing to the multi-level menu will be formed at the front of the image formed by the different input media used by the operator in the simulated scene. The operator determines the position of the triggerable object through the ray, and triggers the triggerable object through a click operation.

It can be seen that, in the related art, whether by forming a pointing point at the front end of the hand image or the operation handle image, or by forming a ray pointing to the multi-level menu at the front end of the hand image or the operation handle image, there are different Stability, this kind of instability is easy to accidentally touch other triggerable objects in the multi-level menu when the operator uses a click operation to select the triggerable object. When the operator interacts with the triggerable object through the pointing point, the instability of false touch may be caused by the area of the pointing point. As shown in FIG. 2 , the operating object 2024 is the target triggerable object of the operator 102, However, since the pointing point 206 has a certain area, and in the multi-level menu 202, the operation objects are closely arranged, when the pointing point 206 is close to the operation object 2024, part of the area of the pointing point 206 will fall into the area corresponding to the operation object 2024. Next to the scope of the operand 2026. At this time, when the operator 102 uses a click operation, the operation object 2026 will be touched by mistake. When the operator interacts with the triggerable object through the ray, the instability of the false touch can be due to the far-field interaction, the distance between the hand image or the joystick image and the multi-level menu is too far, the operator is manipulating the ray When the operator's hand is prone to shaking, it is difficult to accurately determine the position of the target triggerable object at the front end of the ray.

In view of this, an embodiment of the present application provides an interaction method and related equipment based on a multi-level menu, which determines the first operation object by recognizing the first gesture of the first operation object of the multiple operation objects of the multi-level menu type, when the first operation object is an identifier of the first submenu, the first submenu is displayed, and when the first operation object is an identifier of the first operation, the first operation is executed. In this application, gesture recognition is used to complete the interaction, which can reduce the false touch rate of existing interaction methods to a certain extent.

FIG. 3A shows a schematic diagram of an exemplary screen 300 according to an embodiment of the present application.

The screen 300 may be the screen viewed by the operator 102 through the head-mounted wearable device 104. As shown in FIG. 3A, the screen 300 shows a digital world in a simulated scene. A multi-level menu 304 is displayed in the screen 300 .

In some embodiments, multiple triggerable objects may be displayed in the multi-level menu 304, for example, as shown in FIG. After the object is triggered, a corresponding submenu can be generated, or a corresponding operation can be performed (for example, changing a different simulation scene), therefore, these operation objects can be regarded as the triggerable objects.

In some embodiments, the operator 102 can perform gesture input with bare hands, and the system 100 recognizes the input gesture through the collected hand shape image, and then interacts with the multi-level menu 304 based on the recognized gesture.

After the system 100 recognizes the first gesture made by the operator 102 for a certain triggerable object in the multi-level menu 304 , it first determines the type of the triggerable object, so as to execute the corresponding functions of different triggerable objects. As shown in FIG. 3A , in some embodiments, the first gesture may include a pinch gesture 302 based on which the operator 102 interacts with a multi-level menu 304 . The pinching gesture is used to determine the type of the selected target operation object, which conforms to the usage and operation habits of the operator. In some embodiments, the system 100 recognizes that the first finger 3022 and the second finger 3024 of the gesture in the hand image are in contact based on the collected hand image, then the gesture in the hand image can be determined to be the pinch gesture 302 . The fingertips of the first finger 3022 and the second finger 3024 of the pinching gesture 302 can be used to make the operator 102 determine whether to trigger a certain triggerable object in the multi-level menu 304 according to the position of the fingertips.

As shown in FIG. 3A , in some embodiments, the system 100 recognizes that the distance between the pinch gesture 302 made by the operator 102 and the first operation object 3042 is less than or equal to a third distance (for example, 1-2 cm), and may It is determined that the first operation object 3042 is a triggerable object to be triggered by the operator 102 . At this point, the system 100 starts to determine the type of the first operation object 3042 . It can be understood that the setting of the third distance is because if the distance between the pinching gesture 302 and the first operating object 3042 is too far, on the one hand, the system 100 cannot respond to the interaction between the pinching gesture 302 and the first operating object 3042 . On the other hand, the system 100 will recognize the interaction of the pinch gesture 302 with other triggerable objects adjacent to the first operation object 3042 . Therefore, it is necessary to set the distance between the pinching gesture 302 and the first operating object 3042 to be less than or equal to the third distance, so as to accurately respond to the interaction between the pinching gesture 302 and the first operating object 3042 while reducing the false touch rate. In addition, when determining the distance between the pinch gesture 302 and the first operation object 3042, the setting of the reference point of the pinch gesture 302 can be adjusted as required. In some embodiments, the distance between the pinch gesture 302 and the first operation object 3042 may be based on the center of the pinch gesture 302 , and the distance between the base point and the first operation object 3042 is determined. In still some embodiments, the fingertips of the first finger 3022 and the second finger 3024 of the pinch gesture 302 may also be used as a base point to determine the distance between the base point and the first operation object 3042 .

FIG. 3B shows a schematic diagram of an exemplary screen 300 according to an embodiment of the present application.

The pinch gesture 302 is a hand image generated on the screen 300 based on a corresponding gesture made by the operator 102 in the real physical world. In some embodiments, the operator 102 may first make a third gesture in the real physical world. As shown in FIG. 3B , the third gesture includes an open gesture 312 . Based on the hand image generated by the open gesture 312 on the screen 300 , the operator 102 approaches a triggerable object in the multi-level menu. The fingertips of the finger 3122 and the finger 3124 in the open gesture 312 can be used to make the operator 102 determine whether a certain triggerable object in the multi-level menu 304 is a target triggerable object (for example, No. An operation object 3042).

As shown in FIG. 3C , in some embodiments, based on the collected hand shape image, the system 100 recognizes that the opening and closing angle 3126 of the finger 3122 and the finger 3124 of the gesture in the hand image is within a preset angle range (for example, the opening and closing angle 3126 can be set to 0 to 30 degrees), and determine that the gesture in the hand image is the open gesture 312, that is, the initial gesture for making the pinch gesture. The preset angle range can be adjusted based on the operator's operating habits, so as to improve the operator's operating experience. It should be noted that, as an optional implementation manner, the opening and closing angle 3126 may be an acute angle formed with the tiger's mouth of the opening gesture 312 as the center, and the lines connecting the tiger's mouth to the fingertips of the two fingers respectively. The operator 102 keeps the third gesture close to the first operation object 3042 in the multi-level menu 304 until the system 100 recognizes that the distance between the third gesture and the first operation object 3042 is less than or equal to the third distance. On the basis of the opening gesture 312, a pinching gesture 302 is performed on the first operation object 3042, as shown in FIG. 3C.

Based on the implementation of the pinch gesture above, the spread gesture 312 can be set as a gesture for calling out (awakening) a multi-level menu that is conjoined with the pinch gesture 302 . In this way, after the multi-level menu is called out, the operator 102 can continue to make the pinching gesture 302 on the basis of the opening gesture 312 , which saves the energy consumption caused by making other gestures to call out the menu. In some embodiments, the system 100 recognizes the spread gesture 312 made by the operator 102 in the simulated scene, and recognizes that the spread gesture 312 remains at any position for a time greater than or equal to a first time (e.g., hold Stay for 2s), as shown in FIG. 3B , the multi-level menu 304 is displayed at position 1 where the fingertips of the finger 3122 and the finger 3124 of the spread gesture 312 are located. It can be understood that if only the holding time of the open gesture 312 for judging whether to call out the multi-level menu 304 is set equal to the first time, then after the multi-level menu 304 is called out, the system 100 recognizes that the open gesture 312 is still at the same time. If the position stay time exceeds the first time, it is determined that the condition for displaying the multi-level menu 304 is not met, and the multi-level menu 304 will be closed. Therefore, it is necessary to set the hold time of the open gesture 312 to be greater than or equal to the first time, so as to avoid the problem that the multi-level menu 304 is closed due to the gesture being held for too long.

In some other embodiments, the operator 102 can also call out the multi-level menu by touching the head-mounted wearable device 104 . As mentioned above, a collection unit for collecting information may be provided on the head-mounted wearable device 104 , and the collection unit may be a temperature sensor for collecting body surface temperature of the operator 102 (for example, the temperature of the finger of the operator 102 ). The system 100 judges whether the finger of the operator 102 is within the range of touch points set on the head-mounted wearable device 104 for calling out the multi-level menu through the temperature of the finger of the operator 102 collected by the temperature sensor. If the system 100 recognizes that the finger of the operator 102 stays within the range of the touch point for a time greater than or equal to a certain time, a multi-level menu is displayed at any position in the simulation scene.

FIG. 3D shows a schematic diagram of an exemplary screen 300 according to an embodiment of the present application.

The type of the triggerable object may include a submenu or a corresponding operation of the triggerable object. In some embodiments, if the triggerable object selected by the operator 102 is a submenu of the multi-level menu, the operator 102 keeps the pinching gesture moving in the real physical world to control the movement of the triggerable object in the simulated scene. As shown in FIG. 3D , after the system 100 determines that the type of the first operation object 3042 is the logo 3044 of the first submenu, if it recognizes that the operator 102 maintains the pinching gesture 302 to move in the direction 1 in the simulated scene, the system 100 controls the first submenu. A submenu icon 3044 moves in the simulated scene following the pinch gesture 302 . In some embodiments, the image of the logo 3044 of the first submenu in the simulated scene can be different from the image transparency of the first submenu, and can also be different from the size of the first submenu, and can also be a module image of a single color Presented in simulated scenes, etc.

In some embodiments, during the movement of the sign 3044 of the first submenu following the pinch gesture 302, the operator 102 can determine the position of the pinch gesture 302 based on the hand image in the simulated scene, and then according to the first pinch gesture 302 The position where the fingertips of the first finger 3022 and the second finger 3024 are located, or the position where the first submenu is displayed is determined according to the position where the first submenu mark 3044 is located. As shown in FIG. 3E , if the operator 102 determines to display the first submenu 306 at position 2, after the operator 102 determines that the logo 3044 of the first submenu is located at position 2 according to the image in the simulated scene, in the real physical world based on pinching Gesture 302 makes a second gesture, as shown in Figure 3F. The second gesture includes the release gesture 322. Based on the pinch gesture 302, the release gesture 322 is set as a determination condition for displaying the first submenu, which conforms to the usage and operation habits of the operator, and improves the operator's operating experience. After the system 100 recognizes the release gesture 322 made by the operator 102 , it displays the image of the first submenu 306 at position 2 . As shown in FIG. 3F , in some embodiments, based on the collected hand shape image, the system 100 recognizes that the opening and closing angle 3226 of the first finger 3222 and the second finger 3224 of the gesture in the hand shape image is within a preset angle range (for example , the angle range of the opening and closing angle 3226 can be set to be the same as the angle range of the opening and closing angle 3126 ), and the gesture in the hand image is determined to be the release gesture 322 .

In order to avoid the problem that the first submenu 306 cannot be successfully displayed due to the distance between the first submenu mark 3044 and the multi-level menu 304 being too close, as shown in FIG. 3G , in some embodiments, it is necessary to set the first After the mark 3044 of a submenu moves a distance 31 following the pinch gesture 302, or the distance between the mark 3044 of the first submenu and the multi-level menu 304 exceeds a first distance (for example, 1-2 cm), the system 100 may identify The first submenu 306 is displayed after releasing the gesture 322 . During the process that the operator 102 uses the hand image generated in the simulation scene to control the movement of the logo 3044 of the first submenu, in order to enable the operator 102 to determine the logo 3044 of the first submenu and the multi-level menu according to the image in the simulation scene The distance between 304 has exceeded the first distance. In some embodiments, when the distance between the first submenu's logo 3044 and the multi-level menu 304 is still within the first distance, the logo of the first submenu can be set 3044 and the image of the multi-level menu 304 in the simulated scene are displayed in the same color, or displayed in the same gray scale; when the distance between the first sub-menu's logo 3044 and the multi-level menu 304 exceeds the first distance, set the first sub-menu The menu logo 3044 and the images of the multi-level menu 304 in the simulated scene are restored to their original colors. In this way, through the change of color, the operator 102 can clearly determine the position where the first submenu can be displayed by making the release gesture 322 according to the image in the simulated scene, and will not cause problems due to the logo 3044 of the first submenu and the multi-level menu. 304 are too close together to successfully display the first submenu 306 .

In some embodiments, when the operator 102 selects and moves the icon 3044 of the first submenu based on the image of the pinch gesture 302 generated in the simulated scene, the first submenu can be edited by keeping the pinch gesture 302 mark 3044 to perform operations such as flipping and dragging. As shown in FIG. 3H , the system 100 can control the logo 3044 of the first submenu to move from position 3 to position 4 following the pinch gesture 302, and then the operator 102 can move at position 4 based on the simulated scene according to the viewing angle facing the screen 300. The image of the generated pinching gesture 302 continues to rotate the first submenu logo 3044 in the direction 2, and then, as shown in FIG. 102 Check the content of the target operation object.

FIG. 4A shows a schematic diagram of an exemplary multi-level menu-based interaction screen 400 .

In the related art, after the submenu is selected and displayed by clicking, the multi-level menu is no longer displayed. As shown in FIG. 4A, after the operator 102 clicks and selects the submenu 4026 in the multi-level menu 402 based on the pointing point 406 at the front end of the hand image 404 in the simulated scene, as shown in FIG. 4B, the submenu 4026 will expand to cover more Other triggerable objects in the hierarchy menu 402 . In this way, if the operator 102 does not find the desired function or other triggerable objects in the submenu 4026, he needs to click to return to the previous menu. In the simulated scene, the multi-level menu 402 restores the image shown in FIG. 4A . This is not conducive to the operator 102 finding the target triggerable object, and will waste the time for the operator 102 to find the target triggerable object. Meanwhile, the constant switching between the multi-level menu 402 and different operation objects will increase the energy consumption of the computer.

FIG. 5A shows a schematic diagram of an exemplary screen 500 according to an embodiment of the present application.

To solve the above problem, in some embodiments, it may be set to display the first submenu on the basis of displaying the multi-level menu. In this way, when the operator 102 continues to perform gesture interaction on other operation objects of the multi-level menu, since the multi-level menu is still displayed in the simulation scene, the operator 102 can directly make interactive gestures on other operation objects of the multi-level menu . As shown in FIG. 5A , taking the multi-level menu 502 as an example, in the screen 500 , the first submenu 504 corresponding to the first operation object 5022 of the multi-level menu 502 has been displayed in the simulation scene. After displaying the first submenu 504 and other multiple submenus, the operator 102 continues to make a pinch gesture 508 on the second operation object 5024 of the multi-level menu 502 based on the hand image generated in the simulation scene. The system 100 recognizes the pinching gesture 508 of the operator 102 on the second operation object 5024 of the multi-level menu 502 , and determines that the second operation object 5024 is the target operation object to be selected by the operator 102 . Next, the system 100 determines the type of the second operative object 5024 . If the system 100 determines that the second operation object 5024 is an identifier of the second submenu, it displays the second submenu 506 on the basis of displaying the multi-level menu 502 and the first submenu 504 , as shown in FIG. 5B .

FIG. 5C shows a schematic diagram of an exemplary screen 510 according to an embodiment of the present application.

As shown in FIG. 5C , in some embodiments, taking the multi-level menu 512 as an example, in the screen 510 , the first submenu 514 corresponding to the first operation object 5122 of the multi-level menu 512 has been displayed in the simulation scene. On the basis of displaying the first submenu 514 and other multiple submenus, the operator 102 continues to pinch the first suboperation object 5142 of the operation objects in the first submenu 514 based on the hand image generated in the simulation scene Gesture 518. The system 100 recognizes the pinching gesture 518 of the operator 102 on the first sub-operation object 5142 of the first sub-menu 514 , and determines that the first sub-operation object 5142 is the target operation object to be selected by the operator 102 . Next, the system 100 determines the type of the first sub-operation object 5142 . If the system 100 determines that the first sub-operation object 5142 is an identifier of the third sub-menu, then on the basis of displaying the multi-level menu 512 and the first sub-menu 514, a third sub-menu 516 is displayed as shown in FIG. 5D.

After repeating this operation for many times, multiple submenus displayed in a tree structure as shown in FIG. 5E are formed. When multiple submenus are displayed in the scene with the structure shown in FIG. 5E , the operator 102 can clearly and intuitively see the functions contained in each submenu or the next submenu. When the operator 102 finds the target operation object that he wants to select, he can directly interact with the target operation object through the hand image generated in the simulation scene, avoiding the need to switch between different submenus and return in the related art. Time-consuming and high energy consumption problems. In addition, through such a manner of presenting the submenus, multiple submenus will not overlap, which is beneficial to reduce the difficulty of image recognition of the submenus by the terminal device. It should be noted that the display of multiple menus in a tree structure in the embodiment of the present application is only exemplary, and other display structures that can prevent overlapping or occlusion between multiple menus, or display structures with similar effects are all Should fall within the scope of protection of the embodiments of the present application.

In some embodiments, if the triggerable object selected by the operator 102 is the corresponding first operation of a certain triggerable object in the multi-level menu, the operator 102 keeps the pinch gesture moving in the real physical world to control The identified movement of the first operation in the simulated scene. Similar to the way of controlling the movement of the sign 3044 of the first submenu, as shown in FIG. When moving to direction 1 in the scene, the system 100 controls the logo of the first operation to follow the pinch gesture 302 to move in the simulated scene. Similarly, in some embodiments, the image of the logo of the first operation in the simulated scene may be different from the image transparency of the multi-level menu 304, and may also be different from the size of the multi-level menu 304, and may also be a single-color module Images are rendered in simulated scenes, etc.

In some embodiments, during the movement of the sign of the first operation following the pinch gesture 302, if the first operation needs to be performed at a specific location, the operator 102 may determine the pinch gesture 302 based on the hand image in the simulated scene The position where the first operation is performed is determined according to the positions of the fingertips of the first finger 3022 and the second finger 3024 in the pinch gesture 302 or according to the position of the logo of the first operation. If the first operation is an operation that can be performed at any position in the simulation scene, the operator 102 can make a second gesture at any position based on the pinch gesture 302 in the simulation scene, so that the system 100 performs the first operation. Similar to the manner in which the first sub-menu 306 is displayed, in some embodiments, the second gesture includes a release gesture 322 . After the system 100 recognizes the release gesture 322 made by the operator 102, the first operation is performed.

Likewise, in order to avoid the problem that the system 100 cannot successfully execute the first operation because the distance between the first operation’s logo and the multi-level menu 304 is too close, in some embodiments, it is necessary to set the first operation’s logo to follow After the distance moved by the pinch gesture 302 relative to the multi-level menu 304 exceeds a second distance (for example, 1-2 cm), the system 100 may perform the first operation after recognizing the release gesture 322 . During the process that the operator 102 uses the hand image generated in the simulated scene to control the movement of the symbol of the first operation, in order to enable the operator 102 to determine the relationship between the symbol of the first operation and the multi-level menu 304 according to the image in the simulated scene The distance has exceeded the second distance. In some embodiments, when the distance between the first operation's logo and the multi-level menu 304 is still within the second distance, set the first operation's logo and the multi-level menu 304 to simulate The images in the scene are displayed in the same color, or in the same gray scale; when the distance between the logo of the first operation and the multi-level menu 304 exceeds the second distance, set the logo of the first operation and the multi-level menu 304 in the simulated scene The image in is restored to its original color. In this way, through the color change, the operator 102 can clearly determine when to perform the release gesture 322 to make the system 100 perform the first operation according to the images in the simulated scene, and will not cause problems due to the identification of the first operation and the multi-level menu. 304 are too close for the system 100 to successfully perform the first operation.

In some embodiments, after the system 100 executes the first operation, other sub-menus or multi-level menus in the simulated scene shown in FIG. 5E will be closed. According to the operator's operating habits, after the first operation is performed, the first operation is considered to be the operator's target operation object, and the multi-level menu is automatically closed at this time, avoiding the need for the operator to manually close the multi-level menu, which is convenient for operation the operator's operation. In addition, by selecting a triggerable object through gesture recognition and determination conditions, it can be recognized whether the gesture has accurately selected a triggerable object, which reduces the false touch rate caused by clicking and selecting without determination conditions in the related art.

FIG. 6A shows a schematic diagram of an exemplary screen 600 according to an embodiment of the present application.

When multiple submenus are displayed in the simulated scene, and the operator 102 wants to close one of the menus, or close all submenus and multi-level menus, in some embodiments, the operator 102 can do it in the real physical world Make a fourth gesture. As shown in FIG. 6A , the fourth gesture includes a palm slap gesture 602 , and the palm slap gesture 602 conforms to the operating habit of the operator and is convenient for the operator to operate. The system 100 closes the corresponding submenu or multilevel menu after the system 100 recognizes that the operator 102 makes a palm slap gesture 602 for the submenu or multilevel menu that wants to be closed. The operator can selectively close a single or multiple menus, which increases the flexibility of operation and improves the user experience of the operator.

The palm slap gesture 602 is not limited to being parallel to the screen 600 as shown in FIG. When the palm slap gesture 602 for the multi-level menu 604 is tilted at a certain angle, the multi-level menu 604 may also be closed. In addition, when the system 100 recognizes that the hand image in the simulated scene moves from the first position 612 to the second position 614, and the distance 616 between the second position 614 and the multi-level menu 604 is less than a third distance (for example, 1-2 cm) , the system 100 may determine that the gesture in the hand image is a palm clapping gesture 602 . It can be understood that the reference point for determining the movement of the palm slap gesture 602 from the first position 612 to the second position 614 is not limited to the heel of the palm as shown in FIG. The center of the palm gesture is used as the reference point, or a certain fingertip of the palm gesture is used as the reference point, which is not limited in this embodiment of the present application. Similarly, the gesture reference point used to determine the distance 616 may also be the aforementioned other position of the palm gesture as the reference point for determining the distance 616 , which is not limited in this embodiment of the present application. In addition, the determination condition used by the system 100 for identifying the palm slap gesture 602 is not limited to the moving distance described in the above-mentioned embodiment. As an optional implementation, the determination condition can also be that the palm slap gesture 602 moves from the first position to 612 moving to the second position 614 and other conditions. At the same time, a certain determination condition or a combination of multiple determination conditions may also be set as the recognition condition of the palm slap gesture 602 to improve the accuracy of gesture recognition, which is not limited in this embodiment of the present application.

As mentioned above, the operator 102 can close a certain menu in the simulated scene, or close all submenus and multi-level menus based on making the palm slap gesture 602 in the simulated scene. As shown in FIG. 6C, in some embodiments, if the operator 102 wants to close a certain menu (for example, submenu 608) in the simulated scene, the operator 102 can only target the submenu based on the hand image in the simulated scene 608 A palm slap gesture 602 is made. System 100 recognizes palm slap gesture 602 for submenu 608 and closes submenu 608 . In some embodiments, if the operator 102 wants to close all the menus in the simulation scene, as shown in FIG. 6A, the operator 102 can make a palm for the overall image composed of multiple menus generated in the simulation scene based on the hand image. Slap gesture 602 . The system 100 recognizes the palm slap gesture 602 for the overall image, and closes all menus in the simulated scene. In still other embodiments, as shown in FIG. 6D , the operator 102 can make a palm slap gesture 602 only for the multi-level menu 606 at the bottom of the tree structure based on the hand image in the simulated scene. Since all menus in the simulation scene are generated based on the multi-level menu 606 , after the system 100 recognizes the palm slap gesture 602 for the multi-level menu 606 , all menus can be closed.

FIG. 7 shows a flowchart of an exemplary multi-level menu-based interaction method 700 according to an embodiment of the present application. The method 700 can be implemented by the system 100, for example, the system 100 generates a picture (for example, the picture 300 of FIG. The hand image generated in the scene makes different gestures to interact with the multi-level menu. In some embodiments, the operator 102 can implement gesture input with bare hands, and the system 100 can collect front images in real time through a camera or camera set in front of the head-mounted wearable device 104, and recognize the images by to recognize the gesture of the operator 102.

Taking any multi-level menu in the simulation scene as an example, the multi-level menu (for example, the multi-level menu 304 in FIG. 3A ) may include multiple operation objects, and the method 700 may include the following steps.

In step S702, in response to recognizing a first gesture for a first operating object of the plurality of operating objects (for example, the first operating object 3042 in FIG. 3A ), the type of the first operating object is determined. The type of the first operation object may include a submenu or an operation corresponding to the first operation object, and different types of operation objects correspond to different display effects.

In some embodiments, in response to recognizing a pinch gesture (eg, pinch gesture 302 of FIG. 3A ) on the first operational object, a type of the first operational object is determined. In some embodiments, determining the first finger (eg, first finger 3022 of FIG. 3A ) and a second finger (eg, second finger 3024 of FIG. 3A ) of the first gesture are in contact with each other in response to recognizing the first gesture. The first gesture is the pinching gesture. The pinching gesture is used to determine the type of the selected target operation object, which conforms to the usage and operation habits of the operator.

In some embodiments, in response to a distance between the first gesture and the first operation object being less than or equal to a third distance, the type of the first operation object is determined. The distance between the pinching gesture 302 and the first operating object 3042 is set to be less than or equal to the third distance, so as to accurately respond to the interaction between the pinching gesture 302 and the first operating object 3042 while reducing the false touch rate.

In some embodiments, before determining the type of the first operation object, the multi-level menu (for example, the multi-level menu in FIG. menu 304). Based on the initial gesture (spread gesture 312 ) of the operator making the pinch gesture, the spread gesture 312 is set to call out the multi-level menu 304 , which saves the energy consumption caused by making other gestures to call out the menu.

In some embodiments, before the type of the first operation object is determined, the multi-level menu is displayed at the position corresponding to the third gesture in response to the third gesture being held for a time greater than or equal to the first time . If only the holding time of the opening gesture 312 used to determine whether to call out the multi-level menu 304 is set equal to the first time, then after the multi-level menu 304 is called out, the system 100 recognizes that the opening gesture 312 still stays at the same position If the time exceeds the first time, it is determined that the condition for displaying the multi-level menu 304 is not met, and the multi-level menu 304 will be closed. Therefore, it is necessary to set the hold time of the open gesture 312 to be greater than or equal to the first time, so as to avoid the problem that the multi-level menu 304 is closed due to the gesture being held for too long.

In step S704, in response to determining that the first operation object is an identifier of the first submenu (for example, the identifier 3044 of the first submenu in FIG. 3D ), display the first submenu (for example, the first submenu in FIG. 3E ). submenu 306).

In some embodiments, on the basis of displaying the multi-level menu (eg, multi-level menu 304 of FIG. 3E ), the first sub-menu (eg, first sub-menu 306 of FIG. 3E ) is displayed. On the basis of simultaneous display of the multi-level menu 304 and the first sub-menu 306, when the operator does not find the target operation object in the first sub-menu 306, he can directly continue to search for other target operation objects for the multi-level menu 304, In this way, the problem of high energy consumption of the computer caused by switching back and forth between menus of different levels to find the target operation object in the related art is avoided.

In some embodiments, in response to determining that the first operation object (for example, the first operation object 3042 in FIG. 3G ) is an identifier of the first submenu (for example, the identifier 3044 of the first submenu in FIG. 3G ), the control The identifier of the first submenu moves following the first gesture (for example, the pinch gesture 302 in FIG. 3G );

In response to a distance (eg, distance 31 in FIG. 3G ) of the first submenu moving following the first gesture being greater than a first distance, the first submenu is displayed. Setting the mark of the first submenu to follow the first gesture to move a distance greater than the first distance and then displaying the first submenu is to avoid that the distance between the mark 3044 of the first submenu and the multi-level menu 304 is too close. The resulting problem that the first submenu 306 cannot be successfully displayed.

In some embodiments, in response to recognizing a second gesture directed to the identification of the first submenu, the first submenu is displayed at a position corresponding to the second gesture. In some embodiments, in response to recognizing an identified release gesture (eg, release gesture 322 of FIG. 3F ) for the first submenu, the first submenu is displayed at a position corresponding to the release gesture. menu. Based on the pinching gesture 302, the releasing gesture 322 is set as a determination condition for displaying the first submenu, which conforms to the usage and operation habits of the operator, and improves the operator's operating experience.

In some embodiments, the opening and closing angles of the first finger (eg, the first finger 3222 of FIG. 3F ) and the second finger (eg, the second finger 3224 of FIG. 3F ) of the second gesture are recognized. (for example, the opening and closing angle 3226 in FIG. 3F ) is within a preset angle range, and it is determined that the second gesture is the release gesture. The preset angle range can be adjusted based on the operator's operating habits, so as to improve the operator's operating experience.

In some embodiments, after displaying the first submenu (for example, the first submenu 504 in FIG. 5A ), in response to identifying a second operation object (for example, the first submenu 504 in FIG. 5A ) for the plurality of operation objects The second operation object 5024) of the first gesture (for example, the pinching gesture 512 in FIG. 5A ), to determine the type of the second operation object;

In response to determining that the second operation object is an identifier of a second submenu, displaying the multilevel menu (for example, the multilevel menu 502 in FIG. 5B ) and the first submenu (for example, the first submenu in FIG. 5B ) Based on the submenu 504), the second submenu (eg, the second submenu 506 of FIG. 5B) is displayed. The multiple displayed menus can be displayed in a tree structure in the simulated scene, so that the operator can clearly and intuitively view the target operation object. At the same time, the multiple menus displayed in the tree structure will not block each other, which is beneficial to reduce the difficulty of image recognition of the submenus by the terminal device. It should be noted that the display of multiple menus in a tree structure in the embodiment of the present application is only exemplary, and other display structures that can prevent overlapping and blocking between multiple menus should fall into the scope of the embodiment of the present application. protected range.

In some embodiments, after displaying the first sub-menu (for example, the first sub-menu 514 of FIG. 5C ), in response to identifying the first sub-operation object for the plurality of sub-operation objects (for example, The first gesture of the first sub-operation object 5142) (for example, the pinching gesture 518 in FIG. 5C ), to determine the type of the first sub-operation object;

In response to determining that the first sub-operation object is an identifier of a third sub-menu, displaying the multi-level menu (for example, multi-level menu 512 in FIG. 5D ) and the first sub-menu (for example, the first sub-menu in FIG. 5D ) On the basis of a submenu 514), the third submenu (for example, the third submenu 516 of FIG. 5C) is displayed. The multiple displayed menus can be displayed in a tree structure in the simulated scene, so that the operator can clearly and intuitively view the target operation object. At the same time, the multiple menus displayed in the tree structure will not block each other, which is beneficial to reduce the difficulty of image recognition of the submenus by the terminal device. It should be noted that the display of multiple menus in a tree structure in the embodiment of the present application is only exemplary, and other display structures that can prevent overlapping or occlusion between multiple menus, or display structures with similar effects are all Should fall within the scope of protection of the embodiments of the present application.

In some embodiments, after displaying the first submenu, in response to identifying the first submenu (for example, the first submenu 608 of FIG. The fourth gesture (for example, palm slap gesture 602 in FIG. 6A ) of the first submenu 606) closes the first submenu or the multi-level menu. According to the operator's operating habits, it can be set to close a single menu or close all menus at once by closing the initial multi-level menu, increasing the flexibility of operation and improving the user experience of the operator.

In some embodiments, after displaying the first submenu, in response to recognizing a palm slap gesture (eg, palm slap gesture 602 of FIG. 6A ) for the first submenu or the multi-level menu, closing the first submenu or the multi-level menu. The clapping palm gesture 602 conforms to the operating habit of the operator, and is convenient for the operator to operate.

In some embodiments, after the first sub-menu is displayed, in response to recognizing that the palm moves from the first The position (for example, the first position 612 in FIG. 6B ) is moved to a second position (for example, the second position 614 in FIG. 6B ), and the distance between the second position and the first submenu or the multi-level menu (for example, the distance 616 in FIG. 6B ) is smaller than the third distance, and the fourth gesture is determined to be the palm slap gesture. The judging condition of the palm slap gesture 602 is not limited to the moving distance of the palm, but may also be the moving speed of the palm or other judging conditions in line with the operating habits of the operator, which is not limited in this embodiment of the present application.

In step S706, in response to determining that the first operation object is an identifier of the first operation, the first operation is executed.

In some embodiments, in response to determining that the first operation object is an identifier of a first operation, controlling the identifier of the first operation to move following the first gesture;

Performing the first operation in response to the distance that the marker of the first operation moves following the first gesture is greater than a second distance. Setting the mark of the first operation to follow the first gesture to move a distance greater than the second distance before executing the first operation is to avoid the failure of successful execution due to the fact that the mark of the first operation is too close to the multi-level menu The problem with the first operation.

In some embodiments, the first operation is performed in response to recognizing a second gesture identified for the first operation. In some embodiments, the first operation is performed in response to recognizing an identified release gesture for the first operation. Based on the first gesture, setting the second gesture as the determination condition for performing the first operation conforms to the usage and operation habits of the operator, and improves the operator's operating experience.

In some embodiments, in response to recognizing that the opening and closing angles of the first finger and the second finger of the second gesture are within a preset angle range, it is determined that the second gesture is the release gesture. The preset angle range can be adjusted based on the operator's operating habits, so as to improve the operator's operating experience.

In some embodiments, after performing the first operation, the multi-level menu is closed. According to the operator's operating habits, after the first operation is performed, the first operation is considered to be the operator's target operation object, and the multi-level menu is automatically closed at this time, avoiding the need for the operator to manually close the multi-level menu, which is convenient for operation the operator's operation.

It should be noted that the method in the embodiment of the present application may be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario, and is completed by cooperation of multiple devices. In the case of such a distributed scenario, one of the multiple devices may only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete all described method.

It should be noted that some embodiments of the present application are described above. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from those in the above-described embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Based on the same technical idea, and corresponding to the method in any of the above embodiments, the present application also provides an interaction device based on a multi-level menu.

Referring to FIG. 8, the multi-level menu-based interactive device includes:

The recognition module 801 is configured to determine a type of the first operation object in response to recognizing a first gesture for a first operation object of the plurality of operation objects.

In some embodiments, the recognition module 801 is further configured to determine the type of the first operation object in response to recognizing the pinch gesture on the first operation object.

In some embodiments, the recognition module 801 is further configured to determine that the first gesture is the pinch gesture in response to the first finger being recognized as being in contact with the second finger of the first gesture.

In some embodiments, the recognition module 801 is further configured to determine the type of the first operation object in response to a distance between the first gesture and the first operation object being less than or equal to a third distance.

In some embodiments, the recognition module 801 is further configured to display the multi-level menu in response to recognizing the third gesture.

An execution module 802 configured to display the first submenu in response to determining that the first operation object is a first submenu; or

In response to determining that the first operation object is an identifier of a first operation, the first operation is executed.

In some embodiments, the execution module 802 is further configured to display the first submenu on the basis of displaying the multi-level menu.

In some embodiments, the executing module 802 is further configured to, in response to determining that the first operation object is an identifier of a first submenu, control the identifier of the first submenu to move following the first gesture;

The first submenu is displayed in response to the distance that the symbol of the first submenu moves following the first gesture is greater than a first distance.

In some embodiments, the execution module 802 is further configured to display the first submenu at a position corresponding to the second gesture in response to recognizing a second gesture directed to the identification of the first submenu .

In some embodiments, the execution module 802 is further configured to display the first submenu at a position corresponding to the release gesture in response to recognizing a release gesture for the identification of the first submenu.

In some embodiments, the executing module 802 is further configured to, in response to determining that the first operation object is an identifier of a first operation, control the identifier of the first operation to move following the first gesture;

Performing the first operation in response to the distance that the marker of the first operation moves following the first gesture is greater than a second distance.

In some embodiments, the executing module 802 is further configured to execute the first operation in response to recognizing the identified second gesture for the first operation.

In some embodiments, the executing module 802 is further configured to execute the first operation in response to recognizing the identified release gesture for the first operation.

In some embodiments, the execution module 802 is further configured to determine that the second gesture is The release gesture.

In some embodiments, the execution module 802 is further configured to close the multi-level menu.

In some embodiments, the executing module 802 is further configured to determine the type of the second operating object in response to recognizing the first gesture for the second operating object of the plurality of operating objects;

In response to determining that the second operation object is an identifier of a second submenu, the second submenu is displayed on the basis of displaying the multi-level menu and the first submenu.

In some embodiments, the execution module 802 is further configured to determine the type of the first sub-operation object in response to recognizing the first gesture directed at the first sub-operation object of the plurality of sub-operation objects;

In response to determining that the first sub-operation object is an identifier of a third sub-menu, the third sub-menu is displayed on the basis of displaying the multi-level menu and the first sub-menu.

In some embodiments, the execution module 802 is further configured to display the multi-level menu at the position corresponding to the third gesture in response to the third gesture being held for a time greater than or equal to the first time.

In some embodiments, the execution module 802 is further configured to close the first submenu or the multilevel menu in response to recognizing a fourth gesture directed at the first submenu or the multilevel menu.

In some embodiments, the execution module 802 is further configured to close the first submenu or the multilevel menu in response to recognizing a palm slap gesture for the first submenu or the multilevel menu .

In some embodiments, the execution module 802 is further configured to respond to the recognition that the palm moves from the first position to the second position along the direction close to the first sub-menu or the multi-level menu, and the second The distance between the location and the first submenu or the multi-level menu is less than a third distance, and the fourth gesture is determined to be the palm slap gesture.

For the convenience of description, when describing the above devices, functions are divided into various modules and described separately. Of course, when implementing the present application, the functions of each module can be realized in one or more pieces of software and/or hardware.

The device in the foregoing embodiments is used to implement the corresponding multi-level menu-based interaction method in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same technical concept, and corresponding to the method in any of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor When the program is executed, the multi-level menu-based interaction method described in any one of the above embodiments is realized.

FIG. 9 shows a schematic diagram of a more specific hardware structure of an electronic device provided by this embodiment. The device may include: a processor 1010 , a memory 1020 , an input/output interface 1030 , a communication interface 1040 and a bus 1050 . The processor 1010 , the memory 1020 , the input/output interface 1030 and the communication interface 1040 are connected to each other within the device through the bus 1050 .

The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of this specification.

The memory 1020 may be implemented in the form of ROM (Read Only Memory, read only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, and the like. The memory 1020 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and invoked by the processor 1010 for execution.

The input/output interface 1030 is used to connect the input/output module to realize information input and output. The input/output module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The communication interface 1040 is used to connect a communication module (not shown in the figure), so as to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 1050 includes a path that carries information between the various components of the device (eg, processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device may also include other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

The electronic device in the above embodiments is used to implement the corresponding multi-level menu-based interaction method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same technical concept, and corresponding to the method in any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions use It is used to make the computer execute the multi-level menu-based interaction method described in any one of the above embodiments.

The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, A magnetic tape cartridge, tape disk storage or other magnetic storage device, or any other non-transmission medium, that may be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiments are used to make the computer execute the multi-level menu-based interaction method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

Based on the same inventive concept, and corresponding to the multi-level menu-based interaction method described in any of the above embodiments, the present application further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions may be executed by one or more processors of a computer to cause the computer and/or the processor to execute the multi-level menu-based interaction method. Corresponding to the execution subject corresponding to each step in each embodiment of the multi-level menu-based interaction method, the processor executing the corresponding step may belong to the corresponding execution subject.

The computer program product of the above embodiment is used to enable the computer and/or the processor to execute the multi-level menu-based interaction method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, here No longer.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope of the application (including claims) is limited to these examples; under the thinking of the application, the above embodiments or Combinations of technical features in different embodiments are also possible, steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the application as described above, which are not provided in detail for the sake of brevity.

In addition, for simplicity of illustration and discussion, and so as not to obscure the embodiments of the present application, well-known power/connections associated with integrated circuit (IC) chips and other components may or may not be shown in the provided figures. ground connection. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the implementation of the embodiments of the present application to be implemented. platform (ie, the details should be well within the purview of those skilled in the art). Where specific details (eg, circuits) have been set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that other embodiments may be implemented without or with variations from these specific details. Implement the embodiment of the present application below. Accordingly, these descriptions should be regarded as illustrative rather than restrictive.

Although the application has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of those embodiments will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures such as dynamic RAM (DRAM) may use the discussed embodiments.

The embodiments of the present application are intended to embrace all such alternatives, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. within the spirit and principles of the embodiments of the present application shall be included within the protection scope of the present application.

Claims (24)

1. An interactive method based on a multi-level menu, wherein the multi-level menu comprises a plurality of operation objects, the method comprising:

in response to identifying a first gesture to a first operand of the plurality of operands, determining a type of the first operand;

in response to determining that the first operation object is an identification of a first submenu, displaying the first submenu; or alternatively

And executing the first operation in response to determining that the first operation object is the identification of the first operation.

2. The method of claim 1, wherein the first gesture comprises a pinch gesture, and wherein the determining the type of the first operation object in response to identifying the first gesture for the first operation object of the plurality of operation objects comprises:

in response to identifying a pinch gesture to the first operand, a type of the first operand is determined.

3. The method according to claim 2, wherein the responding to the recognition of the pinch gesture to the first operation object specifically comprises:

in response to identifying that the first finger and the second finger of the first gesture are in contact, determining that the first gesture is the pinch gesture.

4. The method of claim 1, wherein the displaying the first submenu in response to determining that the first operational object is an identification of the first submenu comprises:

and displaying the first submenu on the basis of displaying the multi-level menu.

5. The method of claim 1, wherein the displaying the first submenu in response to determining that the first operational object is an identification of the first submenu comprises:

Controlling the identification of the first submenu to move along with the first gesture in response to determining that the first operation object is the identification of the first submenu;

and displaying the first submenu in response to the identification of the first submenu moving a distance along with the first gesture that is greater than a first distance.

6. The method of claim 5, wherein the displaying the first submenu comprises:

and in response to identifying a second gesture for the identification of the first submenu, displaying the first submenu at a position corresponding to the second gesture.

7. The method of claim 6, wherein the second gesture comprises a unclamp gesture, wherein the displaying the first submenu at a location corresponding to the second gesture in response to identifying the identified second gesture for the first submenu comprises:

and in response to the identification of the release gesture aiming at the identification of the first submenu, displaying the first submenu at a position corresponding to the release gesture.

8. The method of claim 1, wherein the performing the first operation in response to determining that the first operation object is an identification of the first operation comprises:

Responsive to determining that the first operation object is an identification of a first operation, controlling the identification of the first operation to follow the first gesture movement;

and executing the first operation in response to the first operation identification moving along the first gesture by a distance greater than a second distance.

9. The method of claim 8, wherein the performing the first operation comprises:

the first operation is performed in response to identifying a second gesture that is identified for the first operation.

10. The method of claim 9, wherein the second gesture comprises a unclamp gesture, and wherein the performing the first operation in response to identifying the identified second gesture for the first operation comprises:

in response to identifying a release gesture for identification of the first operation, the first operation is performed.

11. The method according to claim 7 or 10, characterized in that the method further comprises:

and determining the second gesture as the loosening gesture in response to the fact that the opening and closing angles of the first finger and the second finger of the second gesture are in a preset angle range.

12. The method of claim 1, wherein the responding to the identification that the first operation object is the first operation, after executing the first operation, comprises:

Closing the multi-level menu.

13. The method of claim 1, wherein the determining the type of the first operation object in response to identifying a first gesture to the first operation object of the plurality of operation objects comprises:

and determining the type of the first operation object in response to the distance between the first gesture and the first operation object being less than or equal to a third distance.

14. The method of claim 4, wherein after displaying the first submenu based on displaying the multi-level menu, the method further comprises:

in response to identifying a first gesture to a second operand of the plurality of operands, determining a type of the second operand;

and in response to determining that the second operation object is the identification of a second submenu, displaying the second submenu on the basis of displaying the multi-level menu and the first submenu.

15. The method of claim 4, wherein the first submenu comprises a plurality of sub-operation objects, and wherein after displaying the first submenu on the basis of displaying the multi-level menu, the method further comprises:

In response to identifying a first gesture to a first sub-operation object of the plurality of sub-operation objects, determining a type of the first sub-operation object;

and in response to determining that the first sub-operation object is the identification of a third sub-menu, displaying the third sub-menu on the basis of displaying the multi-level menu and the first sub-menu.

16. The method of claim 1, wherein, in response to identifying a first gesture to a first operand of the plurality of operands, prior to determining the type of the first operand, the method further comprises:

in response to identifying a third gesture, the multi-level menu is displayed.

17. The method of claim 16, wherein the displaying the multi-level menu in response to identifying a third gesture comprises:

and in response to the third gesture remaining stay for more than or equal to the first time, displaying the multi-level menu at a position corresponding to the third gesture.

18. The method of claim 1, wherein, in response to determining that the first operation object is an identification of a first submenu, after displaying the first submenu, the method further comprises:

Responsive to identifying a fourth gesture directed to the first submenu or the multi-level menu, the first submenu or the multi-level menu is closed.

19. The method of claim 18, wherein the fourth gesture comprises a palm tap gesture, closing the first submenu or the multi-level menu in response to identifying the fourth gesture for the first submenu or the multi-level menu, further comprising:

responsive to identifying a palm tap gesture for the first submenu or the multi-level menu, the first submenu or the multi-level menu is closed.

20. The method of claim 19, wherein responsive to identifying a palm tap gesture for the first submenu or the multi-level menu, specifically comprises:

in response to identifying that the palm is moving from a first position to a second position in a direction proximate to the first submenu or the multi-level menu, and that a distance of the second position from the first submenu or the multi-level menu is less than a third distance, the fourth gesture is determined to be the palm slap gesture.

21. An interactive apparatus based on a multi-level menu, comprising:

An identification module configured to determine a type of a first operation object of the plurality of operation objects in response to identifying a first gesture of the first operation object;

an execution module configured to display a first submenu in response to determining that the first operation object is an identification of the first submenu; or alternatively

And executing the first operation in response to determining that the first operation object is the identification of the first operation.

22. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 20 when the program is executed by the processor.

23. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 20.

24. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 20.

Images