WO2023001019A1 - Mixed reality apparatus and device, information processing method, and storage medium - Google Patents

Abstract

A mixed reality apparatus and device, an information processing method, and a storage medium. The mixed reality apparatus comprises: a diving information acquisition module, configured to acquire diving information; a processing module, configured to implement a function of any one of an augmented reality (AR) diving mode and a virtual reality (VR) diving mode on the basis of the diving information, the AR diving mode comprising: one or more of a first functional mode for monitoring whether an underwater environment where a user is located is in an abnormal state, a second functional mode for monitoring whether the user is in an abnormal state, a third functional mode for presenting introductory information of a gazed object, and a fourth functional mode for initiating a diving interaction, and the VR diving mode comprising: one or more of a fifth functional mode for responding to the diving interaction and a sixth functional mode for presenting a diving trajectory; and a display module, configured to display any one of an AR screen and a VR screen.

Description

Mixed reality device and equipment, information processing method and storage medium

This application claims the priority of the Chinese patent application submitted to the China Patent Office on July 23, 2021, with the application number 202110836392.5, and the title of the invention is "mixed reality device and equipment, information processing method and storage medium", the content of which should be understood as Incorporated into this application by reference.

technical field

Embodiments of the present disclosure relate to but are not limited to the technical field of information processing, and in particular, relate to a mixed reality device and device, an information processing method, and a storage medium.

Background technique

Divers usually carry diving equipment when conducting diving activities such as sightseeing, surveying, salvage, repairing and underwater engineering in the underwater environment. However, due to factors such as single function and insufficient intelligence of the diving equipment used in the underwater environment , making most of the public unable to feel the fun of diving activities, which is not conducive to promoting the development and progress of diving activities. Therefore, it is necessary to provide an intelligent diving device with rich functions.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

In the first aspect, the embodiment of the present disclosure provides a mixed reality device, including: a processing module, a display module and a diving information collection module, wherein,

The diving information collection module is configured to collect diving information, and send the diving information to the processing module; wherein, the diving information includes: the user's head information, the user's eye information and the user's location One or more of the environmental information of the underwater environment;

The processing module is configured to implement any function in the augmented reality AR diving mode and the virtual reality VR diving mode based on the diving information; wherein, the AR diving mode includes: for monitoring the underwater environment where the user is located. The first function mode is whether there is an abnormal state in the lower environment, the second function mode is used to monitor whether the user has an abnormal state, the third function mode is used to display the introduction information of the gaze object, and the fourth function mode is used to initiate diving interaction One or more of them, the VR diving mode includes: one or more of a fifth function mode for responding to diving interactions and a sixth function mode for displaying diving tracks;

The display module is configured to display any one of augmented reality images and virtual reality images.

In a second aspect, an embodiment of the present disclosure provides an information processing method, which is applied to the mixed reality device described in the above embodiment, the method includes: acquiring diving information through a diving information collection module, and the diving information includes: user One or more of the head information of the user, the user's eye information and the environmental information of the underwater environment in which the user is located; based on the diving information, any one of the augmented reality AR diving mode and the virtual reality VR diving mode can be realized. wherein, the AR diving mode includes: a first function mode for monitoring whether an abnormal state occurs in the underwater environment in which the user is located, a second function mode for monitoring whether the user has an abnormal state, and a display for displaying One or more of the third functional mode of gazing at the introduction information of the object and the fourth functional mode for initiating diving interaction, and the VR diving mode includes: a fifth functional mode for responding to diving interaction and for displaying One or more of the sixth functional modes of the diving track; the display module is controlled to display any one of augmented reality images and virtual reality images.

In a third aspect, an embodiment of the present disclosure provides a mixed reality device, including: a processor and a memory storing a computer program that can run on the processor, wherein the above embodiments are implemented when the processor executes the program The steps of the information processing method described in.

In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the information processing method described in the above-mentioned embodiments A step of.

Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure. Other advantages of the present disclosure can be realized and obtained through the solutions described in the specification and the accompanying drawings.

Other aspects will be apparent to others upon reading and understanding the drawings and detailed description.

Description of drawings

The accompanying drawings are used to provide an understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used together with the embodiments of the present disclosure to explain the technical solutions of the present disclosure, and do not constitute limitations to the technical solutions of the present disclosure. The shape and size of each component in the drawings do not reflect true scale, but are for the purpose of schematically illustrating the present disclosure.

FIG. 1 is a schematic structural diagram of a mixed reality diving system in an exemplary embodiment of the present disclosure;

FIG. 2 is a schematic structural diagram of a mixed reality device in an exemplary embodiment of the present disclosure;

FIG. 3 is a schematic flowchart of an information processing method in an exemplary embodiment of the present disclosure;

Fig. 4 is a schematic structural diagram of a mixed reality device in an exemplary embodiment of the present disclosure.

detailed description

Several embodiments are described herein, but the description is exemplary rather than limiting, and many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the exemplary embodiments, many other combinations of the disclosed features are possible. Except where expressly limited, any feature or element of any embodiment may be used in combination with, or substituted for, any other feature or element of any other embodiment.

In describing representative embodiments, the specification may have presented a method and/or process as a particular sequence of steps. However, to the extent the method or process is not dependent on a particular order of the steps herein, the method or process should not be limited to the particular order of the steps. Other sequences of steps are also possible, as will be appreciated by those of ordinary skill in the art. Therefore, the specific order of the steps set forth in the specification should not be construed as limitations on the claims. Furthermore, claims to the method and/or process should not be limited to performing their steps in the order written, as those skilled in the art can readily appreciate that such order can be varied and still remain within the spirit and scope of the disclosed embodiments Inside.

In the drawings of the present disclosure, the size of each component, the thickness of a layer, or a region is sometimes exaggerated for the sake of clarity. Therefore, one aspect of the present disclosure is not necessarily limited to the size, and the shape and size of each component in the drawings do not reflect the true scale. In addition, the drawings schematically show ideal examples, and one aspect of the present disclosure is not limited to shapes, numerical values, and the like shown in the drawings.

In the exemplary embodiments of the present disclosure, ordinal numerals such as "first", "second" or "third" are provided to avoid confusion of constituent elements, rather than to limit in terms of quantity.

In the exemplary embodiments of the present disclosure, for convenience, "middle", "upper", "lower", "front", "rear", "vertical", "horizontal", "top", "bottom" are used ", "inner" or "outer" and other words indicating orientation or positional relationship are used to illustrate the positional relationship of constituent elements with reference to the drawings, which are only for the convenience of describing this specification and simplifying the description, rather than indicating or implying the referred device or element Having a particular orientation, being constructed in a particular orientation, and operating in a particular orientation, therefore, are not to be construed as limitations of the present disclosure. The positional relationship of the constituent elements changes appropriately according to the direction in which each constituent element is described. Therefore, it is not limited to the words and phrases described in the specification, and may be appropriately replaced according to circumstances.

In the exemplary embodiments of the present disclosure, the terms "installation", "connection" or "connection" should be interpreted in a broad sense unless otherwise clearly specified and limited. For example, it may be a fixed connection, or a detachable connection, or an integral connection; it may be a mechanical connection, or an electrical connection; it may be a direct connection, or an indirect connection through an intermediate piece, or an internal communication between two components. Those of ordinary skill in the art can understand the meanings of the above terms in the present disclosure according to the actual situation.

In the exemplary embodiments of the present disclosure, the term "module" used may refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware and/or software codes capable of executing and The function associated with this element.

In the exemplary embodiments of the present disclosure, the terms "interface" and "user interface" used may refer to a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the internal form of information and the user can Accepts conversions between forms. The commonly used form of user interface is the graphical user interface (Graphic User Interface, GUI), which refers to the user interface related to the operation displayed in a graphical way. The user interface may include visual interface elements such as icons, windows, buttons, and dialog boxes.

Mixed reality (Mixed Reality, MR) technology is actually a combination of augmented reality (Augmented Reality, AR) technology and virtual reality (Virtual Reality, VR) technology. Using MR technology, users can see the real world (a feature of AR technology) and at the same time see virtual objects (a feature of VR technology). Therefore, MR technology is the development of virtual reality technology. MR technology can introduce real scene information into the virtual environment and set up an interactive feedback information loop between the virtual world, the real world and users, which can enhance the sense of reality of user experience. , has the characteristics of authenticity, real-time interaction and conception.

An embodiment of the present disclosure provides a mixed reality diving system. In practical applications, the mixed reality diving system can be used in diving activities such as sightseeing, survey, salvage, repair and underwater engineering.

FIG. 1 is a schematic structural diagram of a mixed reality diving system in an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the mixed reality diving system may include: a terminal 11 and N mixed reality devices, wherein the mixed reality device and the terminal 11 Communication connection is possible. N is a positive integer greater than or equal to 1. For example, as shown in FIG. 1 , the N mixed reality devices may include: mixed reality device 121 , mixed reality device 122 , . . . , and mixed reality device 12N.

In an exemplary embodiment, the terminal may be an electronic device such as a server, a smart phone, a tablet computer, a notebook computer, or a desktop computer. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, taking the terminal as an example, the server is configured to process and feed back one or more types of information transmitted by the processing module of the mixed reality device, and feed back to the processing module of the mixed reality device at the same time content to be displayed. For example, the server is configured to process the environmental information, so as to determine whether the underwater environment in which the user is in an abnormal state based on the environmental information, so as to generate and issue a warning message when the abnormal state occurs; or, when the diver is in an abnormal state, Send the user's location information to rescuers, so that rescuers can rescue divers who are in an abnormal state. The information processed by the server is different according to the function of the diving mode limited by the mixed reality device. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the terminal may have multiple graphics card ports, each mixed reality device may be communicatively connected to the terminal through one graphics card port, and each graphics card port has a port identifier. For example, the graphics card port may be a high-definition multimedia interface (High Definition Multimedia Interface, HDMI) or a high-definition digital display interface (Display Port, DP). Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the mixed reality diving system may further include: a plurality of wireless signal transmitters, the plurality of wireless signal transmitters are arranged in at least two graphics card ports of the terminal in one-to-one correspondence, and the plurality of mixed reality devices One-to-one wireless communication connection with multiple wireless signal transmitters, so that multiple mixed reality devices are wireless communication connection with the terminal.

For example, the port identifier corresponding to each mixed reality device may be the port identifier of the graphics card port connected to the mixed reality device, or the port identifier of the graphics card port where the wireless signal transmitter connected to the mixed reality device is located.

In an exemplary embodiment, the mixed reality device may be a wearable display device. For example, the wearable display device may include a head-mounted display device or an ear-mounted display device. For example, the wearable display device may be MR diving glasses or an MR diving helmet. Here, the embodiments of the present disclosure do not limit this.

An embodiment of the present disclosure provides a mixed reality device. In practical applications, the mixed reality device can be used in diving activities such as sightseeing, survey, salvage, repair and underwater engineering.

FIG. 2 is a schematic structural diagram of a mixed reality device in an exemplary embodiment of the present disclosure. As shown in FIG. 2 , the mixed reality device 12 may include: a processing module 21, a display module 22 and a diving information collection module 23; wherein, the processing module 21 is connected with display module 22 and diving information collection module 23 respectively;

The diving information collection module 23 is configured to collect diving information, and send the diving information to the processing module 21; wherein, the diving information may include: the user's head information, the user's eye information and the underwater environment where the user is located. One or more of environmental information;

The processing module 21 is configured to implement any function in the augmented reality AR diving mode and the virtual reality VR diving mode based on the diving information; wherein, the AR diving mode may include: for monitoring whether the underwater environment where the user is located appears One of the first function mode for abnormal state, the second function mode for monitoring whether the user has an abnormal state, the third function mode for displaying the introduction information of the gaze object, and the fourth function mode for initiating diving interaction or more, the VR diving mode may include: one or more of a fifth function mode for responding to diving interactions and a sixth function mode for displaying diving tracks;

The display module 22 is configured to display any one of the augmented reality picture corresponding to the AR diving mode and the virtual reality picture corresponding to the VR diving mode.

Here, the user may refer to a diver who wears the mixed reality device and conducts diving activities in an underwater environment.

In this way, the mixed reality device provided by the embodiments of the present disclosure, when the user wears the mixed reality device to carry out diving activities in the underwater environment, collects diving information through the diving information collection module, and based on the collected diving information through the processing module, can Realizing any one of the functions of the AR diving mode and the VR diving mode can realize an intelligent diving device with rich functions, which can facilitate the development and progress of diving activities.

In an exemplary embodiment, the occurrence of an abnormal state in the underwater environment where the user is located may include: in the vicinity of the underwater environment where the user is located (for example, within a preset distance centered on the user's position) There are dangerous objects or dangerous environments that may threaten the safety of users. For example, dangerous objects may include: dangerous animals and plants or obstacles. For example, a hazardous environment may include water velocity exceeding a preset threshold, etc. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the occurrence of an abnormal state by the user may include: the user is in a physical discomfort, for example, the user is in a state of fatigue or fainting. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the gaze object may include: one or more of at least one underwater object in the underwater environment and the underwater environment itself. For example, the introduction information of the gaze object may include: one or more of text, image, video and other information.

In an exemplary embodiment, diving interaction may refer to a diver sharing what he or she sees with one or more other divers (for example, an underwater environment, an object in an underwater environment, etc.), or , may refer to a diver inviting one or more other divers to conduct diving activities together in the same water area, etc. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the diving information collection module may include: a sensor for collecting diving information. For example, the diving information collection module can collect diving information in real time at preset time intervals. For example, the preset time interval may be 1s (second), 2s or 3s, etc. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, taking the diving information including: the user's head information, the user's eye information and the underwater environment where the user is located as an example, as shown in FIG. 2 , the diving information collection module 23 may include: Head information collection module 231, eye information collection module 232 and environment information collection module 233, wherein, head information collection module 231 is configured to collect the user's head information, and send the head information to the processing module 21; The external information collection module 232 is configured to collect the eye information of the user, and sends the eye information to the processing module 21; the environmental information collection module 233 is configured to collect the environmental information of the underwater environment where the user lives, and sends the environmental information to the processing module 21. sent to the processing module 21.

In an exemplary embodiment, the user's head information may include user's head posture information.

In an exemplary embodiment, the head information collection module may include, but is not limited to, an attitude sensor. For example, the attitude sensor is a high-performance three-dimensional motion attitude measuring device based on Micro-Electro-Mechanical System (MEMS) technology, which usually includes a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass, etc. The motion sensor and posture sensor can use these motion sensors to realize the collection of user's head posture information. Of course, the head information collection module can also be implemented by other sensors, which is not limited in this embodiment of the present disclosure.

In an exemplary embodiment, the user's eye information may include: user's eye image information.

In an exemplary embodiment, the eye information collection module may include, but is not limited to, a camera using an image sensor. For example, the camera may be a camera using a complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) image sensor. Of course, the eye information collection module can also be implemented by other sensors, which is not limited in this embodiment of the present disclosure.

In an exemplary embodiment, the environmental information of the underwater environment where the user is located may include: one or more of environmental image information, environmental depth information, and environmental location information of the underwater environment where the user is located. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the environmental information collection module may include, but is not limited to, a camera using an image sensor. For example, the environmental information collection module may be a wide-angle camera (Wide Angle Camera), a fisheye camera (Fisheye Camera), or a depth camera (Deepth Camera), etc. For example, the environmental information collection module may be a CMOS camera or the like. For example, the environmental information collection module may include a first camera for collecting environmental image information of the underwater environment where the user is located and a second camera for collecting environmental depth information of the underwater environment where the user is located. The information acquisition module scans the underwater environment where the user is located, and can collect the environmental image information and environmental depth information of the underwater environment where the user is located, and can perform real-time positioning and map construction (SLAM, Simultaneous Localization And Mapping) . Of course, the environmental information collection module may also include other sensors, for example, a positioning sensor for collecting environmental position information of the underwater environment where the user is located, which is not limited in this embodiment of the present disclosure.

In an exemplary embodiment, the environmental information collection module includes: a first camera for collecting environmental image information of the underwater environment where the user is located; the eye information collection module includes: a first camera for collecting user's eye images The third camera for information and the mixed reality device are head-mounted display devices as an example, the third camera can be set on the inside of the head-mounted display device body, and the first camera can be set on the outside of the head-mounted display device body. When the head-mounted display device body is worn by the user, the first camera can be directed towards the user's eyes, and the third camera can be directed towards the underwater environment where the user is located. Environmental image information for underwater environments.

In an exemplary embodiment, the processing module may include, but not limited to, a central processing unit (Central Processing Unit, CPU), other general purpose processors, a digital signal processor (Digital Signal Processor, DSP), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components or application-specific integrated circuits, etc. The general-purpose processor may be a microprocessor (Micro Processor Unit, MPU) or the processor may be any conventional processor or the like. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the display module may include at least one display module or a device including a display module, for example, a mixed reality display, a head-mounted display, and the like. For example, the display module may be an organic light emitting diode (Organic Light Emitting Diode, OLED) display, a quantum dot light emitting diode (Quantum-dot Light Emitting Diodes, QLED) display, and the like. Here, the embodiments of the present disclosure do not limit this.

Taking the mixed reality diving system including: a server and multiple mixed reality devices as an example, and taking each mixed reality device including: a head information collection module, a head information collection module and an environment information collection module as an example, the exemplary The different working modes of the mixed reality device provided in the embodiments are described in detail.

In an exemplary embodiment, the working modes of the mixed reality device may include: an AR diving mode and a VR diving mode.

The following describes how to switch the working mode of the mixed reality device between the AR diving mode and the VR diving mode.

In an exemplary embodiment, the processing module is configured to acquire the first header information; when the first header information meets the first preset condition, control the display module to display the first confirmation interface (for example, for confirmation Whether to switch the interface of the working mode), and obtain the first eye information; based on the first eye information, determine the user's gaze area on the first confirmation interface; when it is determined that the user's gaze area on the first confirmation interface is the preset first When the area is displayed, switch the working mode from one of the AR diving mode and the VR diving mode to the other of the AR diving mode and the VR diving mode; or, when it is determined that the user's gaze area on the first confirmation interface is the preset In the second display area, keep the working mode unchanged. In this way, the first confirmation interface is triggered to be displayed through the collected head information of the user, and whether to switch the working mode is confirmed through the collected user’s eye information, so that the user can conveniently check the Mixed reality device to operate. Furthermore, the use and operation of the diving device is facilitated, and the convenience of use of the diving device is improved.

In an exemplary embodiment, the first preset condition may be that the user's head is in a head-down state, the user's head is in a head-up state, and the user's head shakes his head one or more times within a preset time (for example, during The shaking amplitude of the user's head in the first direction is greater than the preset first threshold), or, the user's head nods one or more times within the preset time (for example, the shaking of the user's head in the second direction magnitude greater than the preset second threshold), or, within the preset time, the user's head makes a circular movement (for example, the amplitude of the shaking of the user's head in both the first direction and the second direction is greater than the preset third threshold )Wait. For example, when the first head information is the preset head information that indicates that the user performs a head-shaking action within the preset time, that is, the first head information collected within the preset time all indicates that the user moves in the first direction. If the shake of the head is greater than the preset threshold, it can be determined that the first head information meets the first preset condition, and thus, the first confirmation interface can pop up, so that the user can choose whether to switch the working mode through eye movements.

In an exemplary embodiment, the processing module is configured to initialize the mixed reality device, and set the working mode of the mixed reality device to AR diving mode. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the first eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the area of the first confirmation interface located in the line of sight direction as the user's The gaze area of the first confirmation interface.

In an exemplary embodiment, when the working mode is switched from the AR diving mode to the VR diving mode, the first confirmation interface may be a user interface implemented based on AR technology. Or, when switching the working mode from the VR diving mode to the AR diving mode, the first confirmation interface may be a user interface implemented based on VR technology.

The following describes multiple functional modes in which the working mode of the mixed reality device is the AR diving mode.

In an exemplary embodiment, the processing module is configured to obtain the first environmental information in the first functional mode for monitoring whether the underwater environment where the user is in an abnormal state; and send the first environmental information to the server so that the server determines whether the underwater environment in which the user is in an abnormal state based on the first environmental information; receiving the augmented reality screen sent by the server containing warning information indicating that the underwater environment in which the user is in an abnormal state; controlling The display module displays an augmented reality image including warning information. In this way, it is possible to monitor whether there is an abnormal state in the underwater environment where the user is located, and when an abnormal state occurs, the user can be warned by a warning message. Therefore, the personal safety of the divers can be guaranteed, the sense of security of the divers can be improved, and the interest in diving activities can be enhanced.

In an exemplary embodiment, the occurrence of an abnormal state in the underwater environment where the user is located may include: in the vicinity of the underwater environment where the user is located (for example, within a preset distance centered on the user's position) There are dangerous objects or dangerous environments that may threaten the safety of users. For example, dangerous objects may include: dangerous animals and plants, obstacles, and the like. For example, a hazardous environment may include water velocity exceeding a preset threshold, etc. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the warning information may include one or more of information about dangerous objects threatening the user's life in the underwater environment where the user is located and navigation information for instructing the user to travel along the first target route. , wherein the first target route is a route capable of avoiding dangerous objects.

In an exemplary embodiment, as an example, the warning information includes information about dangerous objects threatening the user's life safety in the underwater environment where the user is located and navigation information for instructing the user to travel along the first target route, then the processing The module is also configured to control the display module to display the augmented reality picture containing the information of the dangerous object, and obtain the second eye information; based on the second eye information, determine whether the user is watching the information of the dangerous object; when it is determined that the user is watching When information about dangerous objects is displayed, the display module is controlled to display an augmented reality picture including navigation information. In this way, during the user's diving, when dangerous objects appear in the underwater environment where the user is located, the mixed reality device can display the information of the dangerous objects to the user; and after confirming that the user has seen the information of the dangerous objects, display navigation information to the user , so that the user avoids dangerous objects while traveling. In this way, when a dangerous object appears in the underwater environment where the user is located, it can be ensured that the user pays attention to the information of the dangerous object, and the navigation information for avoiding the dangerous object can be displayed to the user. Therefore, the personal safety of the divers in the underwater environment can be effectively guaranteed, the sense of safety of the divers can be improved, and the interest in diving activities can be enhanced.

In an exemplary embodiment, the second eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the area of the first confirmation interface located in the line of sight direction as the user's The gaze area of the first confirmation interface.

In an exemplary embodiment, the processing module is configured to acquire third eye information in the second functional mode for monitoring whether the user is in an abnormal state; When two preset conditions are met, obtain the second environment information; determine the user's location information based on the second environment information; send the user's location information to the server, so that the server sends the user's location information to other mixed reality devices to request Rescue divers using other mixed reality devices. In this way, based on the user's eye information, when it is detected that the diver's time to keep still is abnormal and accompanied by abnormal eye gaze, it can automatically switch to the distress mode, which is convenient for calling rescuers in time, and the user's location information Send it to other mixed reality devices, and send distress signals to surrounding divers for timely rescue, thereby facilitating timely rescue of divers who have accidents, reducing the risk of accidents for divers, reducing the incidence of diving accidents, and improving the safety of diving devices .

In an exemplary embodiment, whether the user is in an abnormal state may include: whether the user is in a physical discomfort, for example, whether the user is in a state of fatigue or fainting. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the second preset condition may be that the user performs the following eye movements, including but not limited to: the user's gaze stays in the same area for more than a preset time, the number of times the user blinks within the preset time is less than The preset threshold, or the user continues to close the eyes within the preset time, etc. For example, taking the collected user's eye information including: the user's eye image information as an example, when it is detected that the eye image information does not include the iris boundary image area including the user's eyeball, it can be determined that the user performs an eye-closing action, Alternatively, when it is detected that the iris area of the user's eyeball gradually decreases and then gradually increases through the multiple pieces of eye image information collected within a preset time, it can be determined that the user blinks.

In an exemplary embodiment, the processing module is further configured to receive the rescuer's information sent by the server; and control the display module to display the rescuer's information.

In an exemplary embodiment, the processing module is configured to obtain the third environment information and the fourth eye information in the third functional mode for displaying the introduction information of the gaze object; based on the fourth eye information, from In the third environment information, the identification information of the user's gaze object in the underwater environment is obtained; the identification information of the gaze object is sent to the server, so that the server sends the introduction information of the gaze object; the introduction information containing the gaze object sent by the server is received and controlling the display module to display the augmented reality screen including the introduction information of the gaze object. In this way, divers can use the eye gaze control to display the introduction information of the gaze object, which is helpful for divers to have a deep understanding of underwater objects and environments.

In an exemplary embodiment, the gaze object may include: one or more of at least one underwater object in the underwater environment and the underwater environment itself. For example, underwater objects may include: animals, plants, or rocks. For example, the underwater environment may include: ocean trenches or volcanoes. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the fourth eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the information of the third environment information located in the line of sight direction as the user's The identification information of the gaze object in the underwater environment.

In an exemplary embodiment, taking the collected environment information including: environment image information as an example, the identification information of the gazing object may include: image information of the gazing object. For example, based on an image recognition algorithm, the image information of the gazing object is extracted from the environment image information, and the image information of the gazing object is determined as the identification information of the gazing object.

In an exemplary embodiment, the introduction information of the gaze object may include: one or more of information such as text, image, and video. For example, taking the target object as an example of a marine animal, the introduction information of the gaze object may include: name, subject, classification, morphological characteristics, living habits, or protection level, etc. Here, the exemplary embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the processing module is configured to acquire the second header information in the fourth functional mode for initiating diving interaction; when the second header information meets the third preset condition, control the display The module displays the second confirmation interface; acquires the fifth eye information; determines the gaze area of the user on the second confirmation interface based on the fifth eye information; when it is determined that the gaze area of the user on the second confirmation interface is the preset third display area , send a request message to the server for requesting diving interaction. In this way, the user can conveniently operate the mixed reality device through the head and eyes, so as to initiate diving interactive activities. Thus, by initiating a diving interactive activity, multiple divers can be used as a whole to facilitate real-time sharing of the pictures they see (for example, the underwater environment they have visited, the animals and plants in the underwater environment they have watched. And then , can enhance the fun of diving.

In an exemplary embodiment, the third preset condition may be that the user's head is in a head-down state, the user's head is in a head-up state, and the user's head shakes his head one or more times within a preset time (for example, during The shaking amplitude of the user's head in the first direction is greater than the preset first threshold), or, the user's head nods one or more times within the preset time (for example, the shaking of the user's head in the second direction magnitude greater than the preset second threshold), or, within the preset time, the user's head makes a circular movement (for example, the amplitude of the shaking of the user's head in both the first direction and the second direction is greater than the preset third threshold )Wait. For example, when the second head information is the preset head information indicating that the user performs a nodding action within the preset time, that is, the first head information collected within the preset time all indicates that the user is in the second direction. If the shake of the user's head is greater than the preset threshold, it can be determined that the second head information meets the third preset condition, and thus, a second confirmation interface can pop up, so that the user can choose to initiate diving interaction through eye movements. For example, when the gaze area of the user's eyes on the second confirmation interface is the preset third display area, it indicates that the user chooses to initiate a diving interactive activity.

In an exemplary embodiment, the second confirmation interface may be a user interface implemented based on AR technology.

In an exemplary embodiment, the fifth eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the area of the second confirmation interface located in the line of sight direction as the user's The gaze area of the second confirmation interface.

In an exemplary embodiment, the processing module is further configured to receive the augmented reality screen containing the information list of divers sent by the server, and control the display module to display the augmented reality screen containing the information list of divers; information; based on the information of the sixth eye, from the information list of divers, obtain the information of the target divers that the user is looking at; send the information of the target divers to the server, so that the server sends the information to the mixed reality device corresponding to the target divers Request message for invitation to dive interaction. In this way, the user can select the target divers to be invited to conduct diving interaction in a targeted manner through eyes.

In an exemplary embodiment, the diver information list may include: information about one or more divers in the underwater environment within a preset distance from the location of the interaction initiator. For example, the diver information in the diver information list may include: location information of the diver or introduction information (for example, name, image information, etc.) of the diver. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the number of target divers may be one or more, for example, two, three, four, and so on. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the request message for inviting diving interaction may include: introduction information of the object to be shared, and the object to be shared includes: the underwater environment of the interaction initiator and the gaze of the interaction initiator One or more of the underwater objects found, the introduction information of the object to be shared includes: one or more of text, image and video. In this way, multiple divers can share what they see in real time as a whole (for example, the underwater environment they have visited, the animals and plants in the underwater environment they have seen), which can enrich diving activities and enhance diving experience .

In an exemplary embodiment, the processing module is further configured to receive the augmented reality screen containing the position information of at least one diver among the target divers sent by the server; and control the display module to display the augmented reality screen containing the position information of at least one diver. Augmented reality screen. In this way, it is convenient for the initiator of the diving interaction to know the location information of the receiver of the diving interaction, which can enhance the fun of diving.

In an exemplary embodiment, the processing module is further configured to receive an augmented reality screen containing the updated position information of at least one diver; and control the display module to display the augmented reality screen containing the updated position information of at least one diver. realistic picture. In this way, during the process of waiting for the diving interaction recipient, the location of the diving interaction recipient can be updated in real time.

In an exemplary embodiment, the sixth eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; The information of the target diver who is determined to be the user's gaze.

In an exemplary embodiment, the processing module is further configured to display the third confirmation interface when it is determined that the gaze area of the user on the second confirmation interface is the preset fourth display area; acquire seventh eye information; based on The seventh eye information determines the gaze area of the user on the third confirmation interface; when it is determined that the gaze area of the user on the third confirmation interface is the preset fifth display area, the current working mode ends. In this way, the user can conveniently operate the mixed reality device through the head and eyes, so as to end the current working mode, for example, end the diving interactive activity.

In an exemplary embodiment, the third confirmation interface may be a user interface implemented based on AR technology.

In an exemplary embodiment, the seventh eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the area of the third confirmation interface located in the line of sight direction as the user's The third confirms the gaze area of the interface.

The working mode of the mixed reality device is described below as a plurality of functional modes of the VR diving mode.

In an exemplary embodiment, the processing module is configured to control the display module to display a fourth confirmation in response to a request message sent by the server for confirming whether to perform diving interaction in the fifth functional mode for responding to diving interaction interface; obtain eighth eye information; determine the gaze area of the user on the fourth confirmation interface based on the eighth eye information; when it is determined that the gaze area of the user on the fourth confirmation interface is the preset sixth display area, the receiving server sends The virtual reality screen containing the introduction information of the object to be shared; the display module is controlled to display the virtual reality screen containing the introduction information of the object to be shared. In this way, when receiving the diving interaction invitation, the fifth confirmation interface can be displayed, so that the user can conveniently operate the fifth confirmation interface through eyes to determine whether to accept the diving interaction. In this way, by accepting diving interactive activities, multiple divers can share the pictures they see in real time (for example, the underwater environment they have visited, the animals and plants in the underwater environment they have watched. Furthermore, the fun of diving can be enhanced. Moreover, displaying the introduction information of the object to be shared through VR technology can enhance the user's diving experience by introducing virtual underwater environment information into the real underwater environment.

In an exemplary embodiment, the object to be shared includes: one or more of the underwater environment of the initiator of the diving interaction and the underwater object that the initiator of the interaction is watching, and the introduction of the object to be shared The information includes: one or more of text, image and video. In this way, multiple divers can share what they see in real time as a whole (for example, the underwater environment they have visited, the animals and plants in the underwater environment they have seen), which can enrich diving activities and enhance diving experience .

In an exemplary embodiment, the fourth confirmation interface may be a user interface implemented based on VR technology.

In an exemplary embodiment, the processing module is further configured to send a response message for accepting diving interaction to the server, so that the server sends the second navigation information for instructing the user to travel along the second target route; receiving the message sent by the server the virtual reality screen containing the second navigation information, and control the display module to display the virtual reality screen containing the second navigation information, wherein the second target route is a route from the position of the diving interaction acceptor to the position of the diving interaction initiator. In this way, during the diving interaction, by displaying the virtual reality screen containing the second navigation information, the recipient of the diving interaction can perform contact interaction according to traveling to the underwater environment where the initiator of the diving interaction is located, thereby enhancing the fun of diving. Moreover, it helps the diving interaction recipient to quickly enter the underwater environment where the diving interaction initiator is located, saving a lot of time and cost.

In an exemplary embodiment, the processing module is further configured to acquire the third header information; when the third header information meets the fourth preset condition, control the display module to display the fifth confirmation interface; acquire the ninth-eye internal information; based on the ninth eye information, determine the gaze area of the user on the fifth confirmation interface; when it is determined that the gaze area of the user on the fifth confirmation interface is the preset seventh display area, send a response message to the server to accept diving interaction , so that the server sends the second navigation information for instructing the user to travel along the second target route. In this way, the user can control whether to send a response message to the server to accept diving interaction through the head and eyes.

In an exemplary embodiment, the fourth preset condition may be that the user's head is in a head-down state, the user's head is in a head-up state, and the user's head shakes his head one or more times within a preset time (for example, during The shaking amplitude of the user's head in the first direction is greater than the preset first threshold), or, the user's head nods one or more times within the preset time (for example, the shaking of the user's head in the second direction magnitude greater than the preset second threshold), or, within the preset time, the user's head makes a circular movement (for example, the amplitude of the shaking of the user's head in both the first direction and the second direction is greater than the preset third threshold )Wait. For example, when the third header information is preset header information indicating that the user has performed a nodding action within a preset time, it can be determined that the third header information meets the fourth preset condition, and thus, the fifth header information can be popped up. Confirmation interface, so that the user can confirm whether to send a response message to the server to accept diving interaction through eye movements.

In an exemplary embodiment, the ninth eye information may be eye image information of the user. For example, the processing module is configured to determine the user's eyeball orientation according to the user's eye image information; determine the user's line of sight direction according to the user's eyeball orientation; determine the area of the fifth confirmation interface located in the line of sight direction as the user's The fifth confirms the gaze area of the interface.

In an exemplary embodiment, the fifth confirmation interface (that is, the interface for confirming whether to travel to the area where the initiator of the diving interaction is located for the diving interaction) may be a user interface implemented based on VR technology.

In an exemplary embodiment, the processing module is configured to obtain fifth environmental information in the sixth functional mode for displaying diving tracks; determine the current location information of the user based on the fifth environmental information; The position information and the historical position information generate a virtual reality picture including the user's diving track; the display module is controlled to display the virtual reality picture including the user's diving track. In this way, the current location information of divers can be obtained, which is convenient for divers to mark the routes they have traveled and special markers. The VR diving mode can narrow the overall view, which helps divers form an overall awareness and make the diving direction clearer. It is oriented and convenient for divers to quickly enter the next diving destination, saving a lot of energy and time for diving.

In an exemplary embodiment, the fifth environment information may be environment location information. In this way, the environmental location information of the underwater environment where the user is currently located may be used as the user's current location information. In this way, the location information of the user is obtained.

The embodiment of the present disclosure also provides an information processing method. It can be applied to the mixed reality device in one or more of the above exemplary embodiments.

Fig. 3 is a schematic flowchart of an information processing method in an exemplary embodiment of the present disclosure. As shown in Figure 3, the information processing method may include:

Step 31: Obtain diving information through the diving information collection module, and the diving information includes: one or more of the user's head information, the user's eye information, and the environmental information of the underwater environment in which the user is located;

Step 32: Based on the diving information, realize the function of any one of the augmented reality AR diving mode and the virtual reality VR diving mode;

Among them, the AR diving mode includes: the first functional mode for monitoring whether the underwater environment where the user is in an abnormal state, the second functional mode for monitoring whether the user is in an abnormal state, and a function mode for displaying the introduction information of the staring object. One or more of the third function mode and the fourth function mode for initiating diving interaction, the VR diving mode includes: the fifth function mode for responding to diving interaction and the sixth function mode for displaying diving tracks one or more of .

Step 33: Control the display module to display any one of augmented reality images and virtual reality images.

In an exemplary embodiment, step 32 may include:

Step 3211: Obtain the first header information;

Step 3212: When the first header information meets the first preset condition, control the display module to display the first confirmation interface, and acquire the first eye information;

Step 3213: Based on the first eye information, determine the gaze area of the user on the first confirmation interface;

Step 3214: When it is determined that the gaze area of the user on the first confirmation interface is the preset first display area, switch the working mode from one of the AR diving mode and the VR diving mode to the other of the AR diving mode and the VR diving mode One; or, when it is determined that the gaze area of the user on the first confirmation interface is the preset second display area, keep the working mode unchanged.

In an exemplary embodiment, step 32 may include:

Step 3221: In the first function mode, obtain the first environment information;

Step 3222: Send the first environmental information to the server, so that the server determines whether the underwater environment where the user is in is abnormal based on the first environmental information;

Step 3223: receiving the augmented reality image sent by the server and including the warning information indicating that the underwater environment where the user is in is abnormal;

Step 3224: Control the display module to display the augmented reality screen containing the warning information.

In an exemplary embodiment, the warning information includes: one or more of information about dangerous objects threatening the user's life in the underwater environment where the user is located and navigation information for instructing the user to travel along the first target route. , wherein the first target route is a route capable of avoiding dangerous objects.

In an exemplary embodiment, step 3224 may include: controlling the display module to display an augmented reality picture containing information about dangerous objects, and acquiring second eye information; based on the second eye information, determining whether the user is looking at the dangerous object information; when it is determined that the user is gazing at the information of the dangerous object, the display module is controlled to display an augmented reality picture containing navigation information.

In an exemplary embodiment, step 32 may include:

Step 3231: In the second function mode, obtain the third eye information;

Step 3232: When the third eye information meets the second preset condition that the user is in an abnormal state, acquire the second environment information;

Step 3233: Determine the location information of the user based on the second environment information;

Step 3234 : Send the user's location information to the server, so that the server sends the user's location information to other mixed reality devices, so as to request rescue from divers using other mixed reality devices.

In an exemplary embodiment, after step 3234, step 32 may further include:

Step 3235: Receive the information of rescuers sent by the server;

Step 3236: Control the display module to display information about rescuers.

In an exemplary embodiment, step 32 may include:

Step 3241: In the third function mode, acquire the third environment information and the fourth eye information;

Step 3242: Based on the fourth eye information, obtain the identification information of the user's gaze object in the underwater environment from the third environmental information;

Step 3243: Send the identification information of the gaze object to the server, so that the server sends the introduction information of the gaze object;

Step 3244: Receive the augmented reality image including the introduction information of the gaze object sent by the server, and control the display module to display the augmented reality image including the introduction information of the gaze object.

In an exemplary embodiment, step 32 may include:

Step 3251: In the fourth function mode, obtain the second header information;

Step 3252: When the second header information meets the third preset condition, control the display module to display the second confirmation interface;

Step 3253: Obtain the fifth eye information; determine the gaze area of the user on the second confirmation interface based on the fifth eye information;

Step 3254a: When it is determined that the gaze area of the user on the second confirmation interface is the preset third display area, send a request message for requesting initiation of diving interaction to the server.

In an exemplary embodiment, after step 3254a, step 32 may further include:

Step 3255a: Receive the augmented reality screen containing the information list of divers sent by the server, and control the display module to display the augmented reality screen containing the information list of divers;

Step 3256a: Obtain the sixth eye information;

Step 3257a: Based on the sixth eye information, obtain the information of the target diver that the user is looking at from the diver information list;

Step 3258a: Send the information of the target diver to the server, so that the server sends a request message for inviting diving interaction to the mixed reality device corresponding to the target diver.

In an exemplary embodiment, the request message for inviting diving interaction includes: introduction information of the object to be shared, the object to be shared includes: one or more of the underwater environment and underwater objects, the object to be shared The introduction information includes: one or more of text, image and video.

In an exemplary embodiment, after step 3258a, step 32 may further include:

Step 3259a: Receive the augmented reality picture including the location information of at least one diver among the target divers sent by the server, and control the display module to display the augmented reality picture including the location information of at least one diver.

In an exemplary embodiment, after step 3253, step 32 may include:

Step 3254b: When it is determined that the gaze area of the user on the second confirmation interface is the preset fourth display area, display the third confirmation interface;

Step 3255b: Obtain the seventh eye information;

Step 3256b: Determine the gaze area of the user on the third confirmation interface based on the seventh eye information;

Step 3257b: When it is determined that the gaze area of the user on the third confirmation interface is the preset fifth display area, end the current working mode.

In an exemplary embodiment, step 32 may include:

Step 3261: In the fifth function mode, in response to the request message sent by the server for inviting diving interaction, control the display module to display the fourth confirmation interface;

Step 3262: Obtain the eighth eye information;

Step 3263: Based on the eighth eye information, determine the gaze area of the user on the fourth confirmation interface;

Step 3264: When it is determined that the user's gaze area on the fourth confirmation interface is the preset sixth display area, receive the virtual reality screen containing the introduction information of the object to be shared sent by the server;

Step 3265: Control the display module to display the virtual reality screen containing the introduction information of the object to be shared.

Wherein, the objects to be shared include: one or more of underwater environments and underwater objects, and the introduction information of the objects to be shared includes: one or more of text, images and videos.

In an exemplary embodiment, after step 3265, step 32 may further include:

Step 3266: Send a response message of accepting diving interaction to the server, so that the server sends the second navigation information for instructing the user to travel along the second target route;

Step 3267: Receive the virtual reality image containing the second navigation information sent by the server, and control the display module to display the virtual reality image containing the second navigation information.

Wherein, the second target route is a route from the location of the diving interaction recipient to the location of the diving interaction initiator.

In an exemplary embodiment, step 3266 may include: acquiring the third header information; when the third header information meets the fourth preset condition, controlling the display module to display the fifth confirmation interface; acquiring the ninth eye information ; Based on the ninth eye information, determine the gaze area of the user on the fifth confirmation interface; when it is determined that the gaze area of the user on the fifth confirmation interface is the preset seventh display area, send a response message accepting diving interaction to the server, to The server is made to issue second navigation information for instructing the user to travel along the second target route. In this way, the user can control whether to send a response message to the server to accept diving interaction through the head and eyes.

In an exemplary embodiment, step 32 may include:

Step 3271: In the sixth function mode, obtain fifth environmental information;

Step 3272: Determine the current location information of the user based on the fifth environmental information;

Step 3273: Based on the user's current location information and historical location information, generate a virtual reality screen including the user's diving track;

Step 3274: Control the display module to display the virtual reality screen including the user's diving track.

The following AR diving mode includes: the first functional mode for monitoring whether the underwater environment where the user is in an abnormal state, the second functional mode for monitoring whether the user is in an abnormal state, and the one for displaying the introduction information of the staring object The third function mode and the fourth function mode for initiating diving interaction, the VR diving mode includes: the fifth function mode for responding to diving interaction and the sixth function mode for displaying diving tracks as an example, taking an exemplary embodiment An application scenario of the above information processing method will be described.

In an exemplary embodiment, the information processing method may include the following processes:

Step 1: The processing module controls the display module to display the initialization setting interface, and the user can operate the initialization setting interface through eyes and select the initialization working mode. For example, the initial working mode can be selected as AR diving mode.

Step 2: The head information collection module collects the first eye information, and sends the first eye information to the head processing module. When the first head information meets the first preset condition (for example, the user's When the shaking amplitude of the head is greater than the preset first threshold), the processing module controls the display module to display the first confirmation interface (that is, the confirmation switching interface), or, when the first head information does not meet the first preset condition, keep the current Operating mode.

Step 3: the eye information collection module acquires the first eye information of the diver; when the processing module determines that the user's gaze area (ie eye gaze information) on the first confirmation interface is the preset first When the area is displayed, the processing module switches the working mode from the AR diving mode to the VR diving mode, and can continue to perform step 20, or, when the processing module determines that the user's gaze area on the first confirmation interface is the preset second display area, Keep the working mode unchanged, and proceed to step 4.

Step 4: The environmental information collection module (for example, including multiple cameras) collects the environmental image information (for example, first environmental information) of the underwater environment where the user is in real time and transmits it to the processing module.

Step 5: The processing module obtains the first environmental information in real time and sends it to the server, and the server returns the processing result of the current environment where the processor is located: when the server calculates that there is a threat to the user in the underwater environment where the user is located based on the first environmental information When there are dangerous objects (such as dangerous creatures or obstacles) that are life-safe, it indicates that the underwater environment where the user is in an abnormal state, then perform step 6, or the processing result of the current environment indicates that the underwater environment where the user is in If there is no abnormal state, go to step 8;

Step 6: The processing module obtains the augmented reality screen sent by the server and contains warning information indicating that dangerous objects that threaten the user's life safety appear in the underwater environment where the user is located. The warning information may include: the location and size of the dangerous object, Controlling the display module to display an augmented reality picture containing warning information for warning reminders;

Step 7: After controlling the display module to display the augmented reality picture containing the information of dangerous objects, the processing module obtains the second eye information collected by the eye information collection module, when the processing module determines that the diver's gaze is on the basis of the second eye information After receiving the information of the dangerous object, the processing module controls the display module to display an augmented reality picture including navigation information. Continue to perform steps 4 to 7 to monitor whether the underwater environment where the user is in is abnormal;

Step 8: The eye information collection module acquires the diver's third eye information and sends it to the processing module;

Step 9: The processing module records the eyeball activity duration data in real time, and when the third eye information meets the second preset condition that the user is in an abnormal state (for example, the user does not blink continuously or does not open the eyes for more than the preset duration), execute Step 10, or, when the third eye information does not meet the second preset condition that the user is in an abnormal state, perform step 13;

Step 10: The environmental information collection module acquires the second environmental information (including environmental location information) and sends it to the processing module, so that the processing module calculates the location information of the user in an abnormal state based on the second environmental information, and sends the user's location information to the server;

Step 11: The server obtains the information list of surrounding divers of the user in an abnormal state, and sends a distress message to the processing module corresponding to the information list of surrounding divers, so as to request other divers to rescue;

Step 12: The processing module receives the rescuer's information sent by the server, controls the display module to display the rescuer's information and distance, and executes step 28;

Step 13: According to the fourth eye information obtained by the eye information collection module and the third environmental information collected by the environmental information collection module, the processing module calculates the identification information of the gaze object that the diver is watching, and obtains the introduction of the gaze object from the server Information, control the display module to display, so that the divers can have a deep understanding of the underwater organisms and the environment;

Step 14: When the second head information obtained by the processing module in the AR diving mode meets the third preset condition (for example, indicating that the diver has performed a nodding action), the processing module controls the display module to display the second confirmation interface (that is, Confirm shared view interface), the user can confirm whether to share the view of the field of view at this time after interacting with the eyes. When it is determined that the gaze area of the user on the second confirmation interface is the preset third display area, and the selection is confirmed to initiate diving interaction to share the view, then step 15 is performed, or, when the processing module determines that the gaze area of the user on the second confirmation interface When the fourth display area is preset, a third confirmation interface is displayed, and step 19 is performed to confirm whether to end the current working mode;

Step 15: The processing module sends a request message to the server for requesting to initiate a diving interaction, and the server responds to the request message and sends the acquired information list of divers in VR diving mode in the diving field to the initiator of the diving interaction Corresponding processing module;

Step 16: The processing module controls the display module to display the augmented reality screen containing the information list of divers. The initiator of the diving interaction can select the diving interaction object through the eyes, and the information of the target divers is sent to the server, and the server will carry out one-on-one or one-on-one Send a request message for inviting diving interaction to many, and accept the feedback result of whether to accept the invitation;

Step 17: After the processing module corresponding to the diving interaction initiator receives the feedback information from the server, it can choose whether to wait for the arrival of the diving companion to conduct diving interaction. When the diving interaction initiator chooses to wait, perform step 18, or, when the diving interaction If the initiator chooses not to wait, go to step 2;

Step 18: The server updates the location information of the divers receiving the invitation in real time, calculates the distance and feeds it back to the processing module of the initiator of the diving interaction, so as to update the display interface;

Step 19: When the processing module determines that the user's gaze area on the third confirmation interface is the preset fifth display area based on the seventh eye information, then perform step 28, or determine that the user's gaze area on the third confirmation interface is not To preset the fifth display area, proceed to step 2;

Step 20: The processing module monitors whether a request message for inviting diving interaction sent by the server is received. When receiving the request message, the processing module controls the display module to display a fourth confirmation interface (for example, an interface for confirming whether to display the introduction information of the object to be shared shared by the initiator of the diving interaction).

Step 21: Based on the acquired eighth eye information, the processing module determines that the gaze area of the user on the fourth confirmation interface is the preset sixth display area, indicating that the user chooses to accept diving interaction, and then receives the information sent by the server containing the object to be shared. Introduce the virtual reality screen of the information; after controlling the display module to display the virtual reality screen of the introduction information containing the object to be shared, perform step 22, or, if the user chooses not to accept, then perform step 24;

Step 22: When the third head information obtained by the processing module in the AR diving mode meets the fourth preset condition (for example, indicating that the diver has performed a nodding action), the processing module controls the display module to display the fifth confirmation interface (for example, , to confirm whether to travel to the area where the initiator of the diving interaction is located to perform the diving interaction interface). After that, interact with the user's eyes to confirm whether to join the contact diving interactive invitation. When the user chooses to join the invitation, then perform step 23, so as to travel to the area where the initiator of the diving interaction is located, or, when the user chooses not to join the invitation, then Execute step 24;

Step 23: The processing module sends a response message of accepting diving interaction to the server, so that the server issues second navigation information for instructing the user to travel along the second target route (for example, including path planning between the initiator and the recipient) . The processing module of the diving interaction recipient receives the virtual reality screen containing the second navigation information sent by the server, and controls the display module of the diving interaction recipient to display the virtual reality screen containing the second navigation information for route prompting;

Step 24: The processing module obtains the fifth environmental information collected by the environmental information collection module and sends it to the server.

Step 25: Based on the fifth environmental information and the pre-stored historical environmental information, the server records all the diver's travel points and marks them as traveled diving tracks, and generates a virtual reality screen containing the user's diving tracks;

Step 26: the server sends the virtual reality screen containing the user's diving track to the processing module;

Step 27: The control display module sent by the processing module server displays a virtual reality screen containing the user's diving track, and then step 2 can be performed;

Step 28: End the dive.

The description of the above method embodiment is similar to the description of the above device embodiment, and has similar beneficial effects as the device embodiment. For the technical details not disclosed in the method embodiments of the present disclosure, those skilled in the art can refer to the descriptions in the device embodiments of the present disclosure to understand, and details are not repeated here.

An embodiment of the present disclosure also provides a mixed reality device, including: a memory and a memory storing a computer program that can run on a processor, wherein, when the processor executes the program, the information processing method in one or more of the above embodiments is implemented A step of.

In an exemplary embodiment, as shown in FIG. 4 , the mixed reality device 40 may include: at least one processor 401; at least one memory 402 and a bus 403 connected to the processor 401; wherein, the processor 401, The memory 402 communicates with each other through the bus 403; the processor 401 is used to call the program instructions in the memory 402, so as to execute the steps of the information processing method in one or more embodiments above.

In an exemplary embodiment, the above-mentioned processor may be a CPU, other general-purpose processors, DSP, field FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or application-specific integrated circuits. A general purpose processor can be an MPU or the processor can be any conventional processor or the like. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, the above-mentioned memory may include a non-permanent memory in a computer-readable storage medium, a random access memory (Random Access Memory, RAM) and/or a non-volatile memory, such as a read-only memory ( Read Only Memory, ROM) or flash memory (Flash RAM), the memory includes at least one memory chip. Here, the embodiments of the present disclosure do not limit this.

In an exemplary embodiment, besides a data bus, the bus may also include a power bus, a control bus, a status signal bus, and the like. However, the various buses are labeled as bus 403 in FIG. 4 for clarity of illustration. Here, the embodiments of the present disclosure do not limit this.

During implementation, the processing performed by the mixed reality device may be completed by an integrated logic circuit of hardware in a processor or instructions in the form of software. That is, the method steps in the embodiments of the present disclosure may be implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software module may be located in storage media such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

Embodiments of the present disclosure further provide a computer-readable storage medium, including a stored program, wherein when the program is running, the touch device where the storage medium is located is controlled to execute the steps of the information processing method in one or more of the above-mentioned embodiments.

In an exemplary embodiment, the above-mentioned computer-readable storage medium may be, for example: ROM/RAM, magnetic disk, optical disk, and the like. Here, this disclosure is not limited in this embodiment.

The above description of the embodiment of the touch control device or the computer-readable storage medium is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For the technical details not disclosed in the embodiments of the touch control device or the computer-readable storage medium of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding. I won't repeat them here.

Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, the functional modules/units in the system, and the device can be implemented as software, firmware, hardware, and an appropriate combination thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components. Components cooperate to execute. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Although the embodiments disclosed in the present disclosure are as above, the above-mentioned content is only an embodiment adopted to facilitate understanding of the present disclosure, and is not intended to limit the present disclosure. Anyone skilled in the art to which this disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in this disclosure, but the scope of patent protection of this disclosure must still be The scope defined by the appended claims shall prevail.

Claims (20)

A mixed reality device, comprising: a processing module, a display module and a diving information collection module, wherein, The diving information collection module is configured to collect diving information, and send the diving information to the processing module; wherein, the diving information includes: the user's head information, the user's eye information and the user's location One or more of the environmental information of the underwater environment; The processing module is configured to implement any function in the augmented reality AR diving mode and the virtual reality VR diving mode based on the diving information; wherein, the AR diving mode includes: for monitoring the underwater environment where the user is located. The first function mode is whether there is an abnormal state in the lower environment, the second function mode is used to monitor whether the user has an abnormal state, the third function mode is used to display the introduction information of the gaze object, and the fourth function mode is used to initiate diving interaction One or more of them, the VR diving mode includes: one or more of a fifth function mode for responding to diving interactions and a sixth function mode for displaying diving tracks; The display module is configured to display any one of augmented reality images and virtual reality images. The device according to claim 1, wherein the diving information collection module comprises: a head information collection module, an eye information collection module and an environment information collection module, wherein, The head information collection module is configured to collect the user's head information; The eye information collection module is configured to collect user's eye information; The environmental information collection module is configured to collect environmental information of the underwater environment where the user is located. The device according to claim 1 or 2, wherein the processing module is configured to acquire first header information; when the first header information meets a first preset condition, control the display module to display The first confirmation interface, and obtain the first eye information; based on the first eye information, determine the gaze area of the user on the first confirmation interface; when it is determined that the gaze area of the user on the first confirmation interface is the predetermined When setting the first display area, switch the working mode from one of the AR diving mode and the VR diving mode to the other of the AR diving mode and the VR diving mode; or, when it is determined that the user When the gaze area of the first confirmation interface is the preset second display area, the working mode remains unchanged. The device according to claim 1 or 2, wherein the processing module is configured to acquire first environment information in the first function mode; send the first environment information to a server, so that the Based on the first environmental information, the server determines whether the underwater environment in which the user is in an abnormal state; receives the augmented reality screen sent by the server and contains warning information indicating that the underwater environment in which the user is in an abnormal state; The display module is controlled to display the augmented reality image containing the warning information. The device according to claim 4, wherein the warning information includes: information about dangerous objects in the underwater environment where the user is located that threatens the user's life and safety, and information in the navigation information used to instruct the user to follow the first target route. One or more, wherein, the first target route is a route capable of avoiding the dangerous object. The device according to claim 5, wherein the processing module is further configured to control the display module to display an augmented reality picture containing information about the dangerous object, and acquire second eye information; based on the first The second eye information is to determine whether the user is gazing at the information of the dangerous object; when it is determined that the user is gazing at the information of the dangerous object, the display module is controlled to display an augmented reality screen containing the navigation information. The device according to claim 1 or 2, wherein the processing module is configured to acquire third eye information in the second function mode; when the third eye information meets the user's abnormal state Under the second preset condition, obtain the second environment information; determine the location information of the user based on the second environment information; send the location information of the user to the server, so that the server sends the location information of the user Send to other mixed reality devices to request rescue from divers using other mixed reality devices. The device according to claim 7, wherein the processing module is further configured to receive the rescuer's information sent by the server; and control the display module to display the rescuer's information. The device according to claim 1 or 2, wherein the processing module is configured to acquire third environment information and fourth eye information in the third function mode; based on the fourth eye information, From the third environment information, obtain the identification information of the user's gaze object in the underwater environment; send the identification information of the gaze object to the server, so that the server sends the introduction information of the gaze object; receive The server sends the augmented reality picture including the introduction information of the gazing object, and controls the display module to display the augmented reality picture including the introduction information of the gazing object. The device according to claim 1 or 2, wherein the processing module is configured to obtain second header information in the fourth function mode; when the second header information meets a third preset condition , control the display module to display the second confirmation interface; acquire the fifth eye information; determine the gaze area of the user on the second confirmation interface based on the fifth eye information; when it is determined that the user is on the second confirmation interface When the gaze area of the user is the preset third display area, a request message for requesting initiation of diving interaction is sent to the server. The device according to claim 10, wherein the processing module is further configured to receive the augmented reality screen containing the information list of divers sent by the server, and control the display module to display the list containing the information of divers Augmented reality screen; obtain the sixth eye information; based on the sixth eye information, from the diver information list, obtain the information of the target diver that the user is watching; send the information of the target diver to to the server, so that the server sends a request message for inviting diving interaction to the mixed reality device corresponding to the target diver. The device according to claim 11, wherein the request message for inviting diving interaction includes: introduction information of the object to be shared, and the object to be shared includes: one or more of the underwater environment and underwater objects The introduction information of the object to be shared includes: one or more of text, image and video. The device according to claim 11, wherein the processing module is further configured to receive an augmented reality picture sent by the server and containing the position information of at least one diver among the target divers; and control the display module An augmented reality frame including location information of the at least one diver is displayed. The device according to claim 10, wherein the processing module is further configured to display the third confirmation interface when it is determined that the gaze area of the user on the second confirmation interface is the preset fourth display area; Seven eye information; based on the seventh eye information, determine the user's gaze area on the third confirmation interface; when it is determined that the user's gaze area on the third confirmation interface is the preset fifth display area, end current working mode. The device according to claim 1 or 2, wherein the processing module is configured to, in the fifth functional mode, control the display module to display The fourth confirmation interface; obtaining the eighth eye information; based on the eighth eye information, determining the gaze area of the user on the fourth confirmation interface; when it is determined that the gaze area of the user on the fourth confirmation interface is preset In the sixth display area, the virtual reality screen containing the introduction information of the object to be shared sent by the server is received; the display module is controlled to display the virtual reality screen containing the introduction information of the object to be shared, wherein the object to be shared includes: One or more of the underwater environment and underwater objects, the introduction information of the object to be shared includes: one or more of text, image and video. The device according to claim 15, wherein the processing module is further configured to send a response message accepting diving interaction to the server, so that the server sends a message for instructing the user to travel along the second target route. Second navigation information: receiving the virtual reality screen containing the second navigation information sent by the server, and controlling the display module to display the virtual reality screen containing the second navigation information, wherein the second The target route is a route from the location of the dive interaction recipient to the location of the dive interaction initiator. The device according to claim 1 or 2, wherein the processing module is configured to obtain fifth environment information in the sixth function mode; and send the fifth environment information to the server, so that the server based on The fifth environmental information and the pre-stored historical environmental information generate a virtual reality screen containing the user's diving track; receive the virtual reality screen containing the user's diving track sent by the server, and control the display module to display the virtual reality screen containing the user's diving track picture. An information processing method, applied to the mixed reality device according to any one of claims 1 to 17, the method comprising: The diving information is acquired through the diving information collection module, and the diving information includes: one or more of the user's head information, the user's eye information and the environmental information of the underwater environment in which the user is located; Based on the diving information, any function in the augmented reality AR diving mode and the virtual reality VR diving mode is realized; wherein, the AR diving mode includes: a first function for monitoring whether an abnormal state occurs in the underwater environment where the user is located One or more of a functional mode, a second functional mode for monitoring whether the user has an abnormal state, a third functional mode for displaying the introduction information of the staring object, and a fourth functional mode for initiating diving interaction, The VR diving mode includes: one or more of a fifth function mode for responding to diving interactions and a sixth function mode for displaying diving tracks; The display module is controlled to display any one of augmented reality images and virtual reality images. A mixed reality device, comprising: a processor and a memory storing a computer program that can run on the processor, wherein the steps of the information processing method according to claim 18 are implemented when the processor executes the program. A computer-readable storage medium, including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the steps of the information processing method as claimed in claim 18 .

Images