[go: up one dir, main page]

WO2024000189A1 - Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage - Google Patents

Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage Download PDF

Info

Publication number
WO2024000189A1
WO2024000189A1 PCT/CN2022/102015 CN2022102015W WO2024000189A1 WO 2024000189 A1 WO2024000189 A1 WO 2024000189A1 CN 2022102015 W CN2022102015 W CN 2022102015W WO 2024000189 A1 WO2024000189 A1 WO 2024000189A1
Authority
WO
WIPO (PCT)
Prior art keywords
yaw direction
deviation
image
mark
identification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/102015
Other languages
English (en)
Chinese (zh)
Inventor
李博文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SZ DJI Technology Co Ltd
Original Assignee
SZ DJI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SZ DJI Technology Co Ltd filed Critical SZ DJI Technology Co Ltd
Priority to CN202280049705.5A priority Critical patent/CN117693663A/zh
Priority to PCT/CN2022/102015 priority patent/WO2024000189A1/fr
Publication of WO2024000189A1 publication Critical patent/WO2024000189A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Definitions

  • the present application relates to the field of control, and in particular, to a control method, a head-mounted display device, a control system and a storage medium.
  • Mobile devices such as unmanned aerial vehicles, underwater/water unmanned robots, ground smart cars, etc.
  • a control terminal such as a remote control
  • the imaging device can capture different images, and the images captured by the imaging device can be transmitted to the display device of the control terminal for user reference, thereby helping the user to further control the movable device.
  • the horizontal movement control of the movable equipment is based on the yaw direction of the head of the movable equipment.
  • the yaw direction of the head of the movable equipment and the yaw direction of the imaging device are Same or roughly the same, the user can know the yaw direction of the aircraft head by observing the image captured by the imaging device, thereby controlling the movable device.
  • the movement direction reference may be different from the yaw direction of the aircraft head.
  • the user can use the head-mounted display device to watch the imaging device mounted on the mobile device capture the image, but cannot know the movement direction reference for horizontal movement control of the movable device when using the control terminal, thus making it impossible for the user to accurately control the movable device.
  • this application provides a control method, a head-mounted display device, a control system and a storage medium, aiming to solve the problem when the movement direction reference of a movable device during horizontal movement control may be different from the yaw direction of the machine head.
  • Technical problems such as the inability to know the horizontal movement reference of the movable device when using the control terminal and the inability to accurately control the movable device.
  • inventions of the present application provide a method for controlling a head-mounted display device.
  • the head-mounted display device is used to display images captured by an imaging device carried by a movable device.
  • the movable device includes a body.
  • the fuselage includes a machine head, and the method includes:
  • an identification mark according to the reference yaw direction and the actual yaw direction, the identification being used to indicate the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction;
  • the reference yaw direction is a movement direction reference for controlling the terminal's horizontal movement control of the movable device.
  • a head-mounted display device including:
  • a display device at least used to display images captured by an imaging device carried by a movable device, the movable device including a body, and the body including a machine head;
  • One or more processors working individually or jointly, to perform the following steps:
  • an identification mark according to the reference yaw direction and the actual yaw direction, the identification being used to indicate the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction;
  • the reference yaw direction is a movement direction reference for controlling the terminal's horizontal movement control of the movable device.
  • control system including:
  • a movable device includes a fuselage, the fuselage includes a machine head;
  • the aforementioned head-mounted display device
  • the head-mounted display device and the control terminal are communicatively connected with the movable device.
  • inventions of the present application provide a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the computer program When executed by a processor, it causes the processor to implement the steps of the above method.
  • Embodiments of the present application provide a head-mounted display device and its control method, storage medium, and storage medium.
  • the method includes: acquiring the reference yaw direction of the head of the movable device; acquiring the actual yaw direction of the imaging device carried by the movable device. Yaw direction; a mark is displayed according to the reference yaw direction and the actual yaw direction, and the mark is used to indicate the reference yaw direction and/or the actual yaw direction and the reference yaw direction.
  • the reference yaw direction is the movement direction reference of the control terminal's horizontal movement control of the movable device, so that the user's movement direction reference when controlling the horizontal movement of the movable device may be consistent with the yaw direction of the machine head
  • at least the reference yaw direction of the movable device can be known through the displayed logo, which facilitates the user to use the control terminal to accurately control the movable device; when the logo is used to indicate that the actual yaw direction is different from the reference
  • it can also facilitate the user to control the actual yaw direction of the imaging device.
  • Figure 1 is a schematic flowchart of a control method for a head-mounted display device provided by an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of a control system in an implementation manner provided by the embodiment of the present application;
  • Figure 3A is a schematic structural diagram of an unmanned aerial vehicle in an implementation manner provided by the embodiment of the present application.
  • Figure 3B is a schematic structural diagram of a gimbal of an unmanned aerial vehicle in an implementation provided by the embodiment of the present application;
  • Figure 4A is a schematic diagram of the attitude of an unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by an embodiment of the present application;
  • Figure 4B is a schematic diagram of displaying a logo on a head-mounted display device according to the reference yaw direction and the actual yaw direction shown in Figure 4A according to an embodiment of the present application;
  • Figure 4C is a schematic diagram of the attitude of another unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by the embodiment of the present application;
  • Figure 4D is a schematic diagram of displaying a logo on a head-mounted display device according to the reference yaw direction and the actual yaw direction shown in Figure 4C according to an embodiment of the present application;
  • Figure 4E is a schematic diagram of the attitude of yet another unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by the embodiment of the present application;
  • Figure 4F is a schematic diagram of displaying a logo on a head-mounted display device according to the reference yaw direction and the actual yaw direction shown in Figure 4E according to an embodiment of the present application;
  • Figure 4G is a schematic diagram of the attitude of yet another unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by the embodiment of the present application;
  • Figure 4H is a schematic diagram of displaying a logo on a head-mounted display device according to the reference yaw direction and the actual yaw direction shown in Figure 4G according to an embodiment of the present application;
  • Figure 5A is a schematic diagram of the attitude of an unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by an embodiment of the present application;
  • Figure 5B is a schematic diagram of displaying another logo on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in Figure 5A according to an embodiment of the present application;
  • Figure 5C is a schematic diagram of the attitude of another unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by the embodiment of the present application;
  • Figure 5D is a schematic diagram of displaying another logo on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in Figure 5C according to an embodiment of the present application;
  • Figure 5E is a schematic diagram of the attitude of yet another unmanned aerial vehicle and its reference yaw direction and actual yaw direction provided by the embodiment of the present application;
  • Figure 5F is a schematic diagram of displaying another logo on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in Figure 5E according to an embodiment of the present application;
  • Figure 6 is a schematic diagram of displaying multiple logos on a head-mounted display device in some implementations provided by the embodiments of this application;
  • Figure 7 is a schematic diagram of displaying yet another logo on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in Figure 4C according to an embodiment of the present application;
  • FIG. 8 is a schematic diagram of displaying yet another logo on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in FIG. 4E according to an embodiment of the present application;
  • Figure 9 is a schematic diagram of another logo displayed on a head-mounted display device based on the reference yaw direction and the actual yaw direction shown in Figure 4E according to an embodiment of the present application;
  • Figure 10 is a schematic diagram of a scene in which the control terminal controls the deflection of the unmanned aerial vehicle in an implementation manner provided by the embodiment of the present application;
  • Figure 11 is a schematic diagram of a scene in which the control terminal controls the deflection of the unmanned aerial vehicle in another implementation manner provided by the embodiment of the present application;
  • Figure 12 is a schematic diagram of a scene in which the control terminal controls the deflection of the unmanned aerial vehicle in yet another implementation manner provided by the embodiment of the present application;
  • Figure 13 is a schematic diagram of the virtual environment of the head-mounted display device in an implementation provided by the embodiment of the present application.
  • Figure 14 is a schematic diagram of a logo displayed in a top view state according to an embodiment of the present application.
  • Figures 15A and 15B are schematic diagrams of scenes of reference yaw direction calibration in an implementation provided by the embodiment of the present application.
  • Figure 16 is a schematic block diagram of a head-mounted display device provided by an embodiment of the present application.
  • Mobile devices such as unmanned aerial vehicles, underwater/water unmanned robots, and ground smart cars, can be equipped with imaging devices for capturing images. Users can control the movable devices through a control terminal (such as a remote control). Control, thereby enabling the imaging device to capture different images.
  • a control terminal such as a remote control
  • the horizontal movement control of the movable equipment is based on the yaw direction of the head of the movable equipment.
  • the yaw direction of the head of the movable equipment and the yaw direction of the imaging device are The same or roughly the same, the user can know the yaw direction of the aircraft nose by observing the image captured by the imaging device.
  • the mobile device can control the machine of the mobile device in response to yaw control instructions sent by at least two control terminals (remote controller and head-mounted display device). The body deflects to adjust the yaw direction of the aircraft nose.
  • the wearable display device changes the horizontal movement direction reference and interferes with the control terminal's control of the movable device.
  • the movement direction reference during horizontal movement control of the movable device may be different from the yaw direction of the aircraft head.
  • the embodiments of the present application mainly use an unmanned aerial vehicle to illustrate the above principles of movable devices.
  • the movement actions in the movable device correspond to the flight actions of the unmanned aerial vehicle, that is, in the specific application scenario of the unmanned aerial vehicle.
  • the moving direction corresponds to the flight direction.
  • the implementation principles are similar and will not be described again.
  • FIG. 1 is a schematic flowchart of a control method for a head-mounted display device provided by an embodiment of the present application.
  • the method can be applied in a head-mounted display device, at least for processes such as displaying signs based on the reference yaw direction of the unmanned aerial vehicle's nose and the actual yaw direction of the imaging device mounted on the unmanned aerial vehicle.
  • unmanned aerial vehicles also called drones
  • the head-mounted display device may include a virtual reality (VR, virtual reality) display device or a first-person perspective (FPV, first person view) display device; of course, it is not limited to this.
  • the head-mounted display device includes Flight control glasses, smart helmets, smart headband devices.
  • the head-mounted display device may be, for example, a glasses-type display device or a helmet-mounted display device.
  • the embodiment of the present application also provides a control system.
  • the control system includes an unmanned aerial vehicle 100, a first control terminal 200 and a second control terminal 300.
  • the unmanned aerial vehicle 100 can communicate with the first control terminal 200 and the second control terminal 300.
  • Two control terminals 300 are connected for communication.
  • the first control terminal 200 and the second control terminal 300 are both used to control the unmanned aerial vehicle 100 .
  • the first control terminal 200 may include at least one of a remote control, a smartphone, and a tablet computer.
  • the second control terminal 300 may include the aforementioned head-mounted display device, and may also include a remote control, a smartphone, or a tablet computer. At least one.
  • Figure 2 is only one possible form of the structure of the control system in an implementation manner provided by the embodiment of the present application.
  • Those skilled in the art can use multiple control terminals to control the unmanned aerial vehicle as needed. It suffices that at least one of the control terminals includes the head-mounted display device in the embodiment of the present application.
  • the unmanned aerial vehicle 100 includes an airframe 110, a power system 120, an imaging device 130, and a control device (not shown in Figure 2).
  • the body 110 may include a nose.
  • the unmanned aerial vehicle 100 further includes an arm, wherein the arm is connected to the fuselage 110 and is used to install the power system.
  • the The power system 120 can be directly installed on the fuselage 110 .
  • the power system 120 is used to provide flight power for the unmanned aerial vehicle.
  • the power system 120 may include a motor and a propeller installed on the motor and driven by the motor.
  • the power system 120 can drive the body 110 of the UAV 100 to rotate around one or more rotation axes.
  • the above-mentioned rotation axis may include a roll axis, a yaw axis, and a pitch axis.
  • the motor can be a DC motor or an AC motor.
  • the motor can be a brushless motor or a brushed motor.
  • the imaging device 130 is carried directly on the fuselage 110 or carried through a pan/tilt, and is used to capture images, and the images may be pictures and/or videos. Due to the different connection methods between the imaging device 130 and the fuselage 110, the implementation of controlling the actual yaw direction of the imaging device will be different, and thus the steps for obtaining the actual yaw direction of the imaging device will be different. Subsequently, imaging will be combined
  • the connection method between the device 130 and the body 110 is different, and it shows how to obtain the actual yaw direction of the imaging device so as to display the corresponding logo on the head-mounted display device, which will not be described again here.
  • the first control terminal 200 may include an input device, wherein the input device may detect a control operation (a pitch control operation and/or a roll control operation) of a user of the first control terminal, and the first control terminal A control instruction (pitch control instruction and/or roll control instruction) to the unmanned aerial vehicle can be generated according to the user's control operation detected by the input device, and the first control terminal can send the control instruction to the unmanned aerial vehicle so that the unmanned aerial vehicle The flight level can be performed according to the control instruction. It can be understood that the first control terminal can also generate the first yaw control instruction according to the yaw control operation of the user of the first control terminal detected by the input device.
  • a control operation a pitch control operation and/or a roll control operation
  • the first control terminal A control instruction (pitch control instruction and/or roll control instruction) to the unmanned aerial vehicle can be generated according to the user's control operation detected by the input device, and the first control terminal can send the control instruction to the unmanned aerial vehicle so that the unmanned aerial vehicle The
  • the first control terminal can Send the first yaw control command to the unmanned aerial vehicle.
  • the input device may include at least one of a button, a joystick, a thumb wheel, and a touch screen.
  • the input device is a joystick, and the joystick is installed on the body of the remote controller.
  • the remote controller senses the user's yaw control manipulation of the joystick and generates a corresponding first yaw control instruction, which is then sent through the communication device of the first control terminal.
  • the information is sent to the unmanned aerial vehicle to control the unmanned aerial vehicle 100 to adjust and control the yaw rotation of the fuselage to adjust the yaw direction of the nose.
  • the second control terminal 300 may include an input device, wherein the input device may detect a control operation of a user of the second control terminal, and the second control terminal may generate a generated signal based on the user's control operation detected by the input device. Control instructions for unmanned aerial vehicles.
  • the second control terminal 300 is provided with a motion sensor, and corresponding control instructions are generated according to the motion of the second control terminal 300 sensed by the motion sensor to control the unmanned aerial vehicle 100 to perform corresponding operations, for example, motion
  • the movement (eg, yaw rotation) of the second control terminal 300 sensed by the sensor generates a second yaw control instruction to control the yaw rotation of the body of the unmanned aerial vehicle 100 to adjust the yaw direction of the nose.
  • the user of the first control terminal may be the same as the user of the second control terminal. In some cases, the user of the first control terminal may be different from the user of the second control terminal.
  • the input device of the second control terminal 300 may be the same as the input device of the first control terminal 200, or may be different.
  • the unmanned aerial vehicle can control the deflection of the body 110 of the unmanned aerial vehicle 100 in response to the yaw control instructions sent by the first control terminal 200 and the second control terminal 300, thereby adjusting the yaw direction of the nose.
  • the horizontal flight control of the UAV 100 is based on the yaw direction of the nose of the UAV 100 as the flight direction reference.
  • both the first control terminal 200 and the second control terminal 300 can change the yaw direction of the nose of the unmanned aerial vehicle 100, the horizontal flight control of the unmanned aerial vehicle by the first control terminal 200 will be controlled by the second control terminal 200.
  • the control terminal 300 interferes because when the second control terminal 300 adjusts the yaw direction of the nose of the unmanned aerial vehicle 100, it will change the flight direction reference of the horizontal flight control of the unmanned aerial vehicle 100 by the first control terminal 200, which will affect the first control terminal 300.
  • the continuous horizontal flight control of the UAV aircraft 100 by the control terminal 200 provides a poor user experience.
  • the yaw rotation of the fuselage 110 is controlled according to the second yaw control instruction and the horizontal flight control of the unmanned aerial vehicle 100 is maintained.
  • the flight direction reference (which can be called the reference yaw direction) remains unchanged, thereby solving the problem that the first control terminal 200 will be interfered by the second control terminal 300 when controlling the horizontal flight of the unmanned aerial vehicle 100 .
  • the second control terminal 300 can receive images captured by the imaging device transmitted by the UAV 100 and display them on a display device.
  • the display device can be integrated with the second control terminal 300 , or the display device can be integrated with the second control terminal 300 .
  • the second control terminal 300 is set up separately and communicates with the second control terminal 300.
  • the communication connection method can be a wired connection method or a wireless connection method, which is not limited here.
  • the second control terminal 300 is the aforementioned head-mounted display device, and the display device of the second control terminal 300 can be integrated on the body of the head-mounted display device, or communicate with the head-mounted display device through a data line.
  • the first yaw control instruction of the first control terminal is used to control the yaw rotation of the fuselage of the unmanned aerial vehicle and update the reference yaw direction to obtain the updated reference yaw direction.
  • the unmanned aerial vehicle may receive a first yaw control instruction generated and sent by the first control terminal, and the unmanned aerial vehicle may control the yaw rotation of the fuselage according to the first yaw control instruction.
  • the unmanned aerial vehicle will update the reference yaw direction according to the first yaw control instruction to obtain the updated reference yaw direction.
  • the unmanned aerial vehicle will update the reference yaw direction according to the first yaw control instruction and the
  • the above-mentioned reference yaw direction determines a new reference yaw direction of the aircraft head, and updates the reference yaw direction with the determined new reference yaw direction of the aircraft nose, where the new reference yaw direction of the aircraft nose is determined.
  • the heading is the updated reference yaw direction of the aircraft nose.
  • the reference yaw direction is updated with the angle of the fuselage yaw rotation corresponding to the first yaw control instruction.
  • the updated reference yaw direction of the aircraft head may be a yaw direction obtained by superposing the angle of the yaw rotation of the fuselage to the reference yaw direction of the aircraft head.
  • the angle of yaw rotation of the fuselage corresponding to the first yaw control instruction may be based on the sensing data of the fuselage output by the sensor of the unmanned aerial vehicle.
  • the senor may include at least one of the following: an inertial measurement unit (Inertial Measurement Unit, IMU for short), a compass and a gyroscope combination unit.
  • IMU Inertial Measurement Unit
  • the sensing data may be used to determine rotation information of the fuselage of the unmanned aerial vehicle in the yaw direction (such as at least one of the actual attitude of the fuselage, the angular velocity and the angular acceleration of the fuselage) .
  • the reference yaw direction can be updated, when the user cannot know the flight direction reference for horizontal flight control of the UAV, that is, the reference yaw direction, it is easy to lose the flight direction reference of the UAV and affect the control of the UAV. .
  • a user watches an image captured by an imaging device carried by an unmanned aerial vehicle through a head-mounted display device, it is easier to lose sight of the flying direction reference of the unmanned aerial vehicle. That is, when the user controls the unmanned aerial vehicle to fly forward, what he or she sees is The picture may not be the image captured by the imaging device while flying forward.
  • Embodiments of the present application may display a logo based on the reference yaw direction of the aircraft head and the actual yaw direction of the imaging device.
  • the logo is used to indicate that the reference yaw direction and/or the actual yaw direction is consistent with the reference.
  • the deviation between the yaw directions allows the user to at least know the reference yaw direction of the unmanned aerial vehicle through the displayed logo when the reference flight direction of the unmanned aerial vehicle is different from the yaw direction of the nose during horizontal flight control, making it easier for the user to use
  • the control terminal controls the unmanned aerial vehicle, such as horizontal flight control of the unmanned aerial vehicle; when the mark is used to indicate the deviation between the actual yaw direction and the reference yaw direction, it can also facilitate the user Control the actual yaw direction of the imaging device.
  • the embodiments of the present application are mainly described by taking the first control terminal as a remote controller and the second control terminal as a head-mounted display device as an example.
  • the control method of the head-mounted display device includes steps S110 to S130.
  • the head-mounted display device is used to display images captured by an imaging device carried by an unmanned aerial vehicle.
  • the unmanned aerial vehicle includes a fuselage, and the fuselage includes a nose.
  • the reference yaw direction is the flight direction reference for the horizontal flight control of the unmanned aerial vehicle by a control terminal, where the control terminal is the aforementioned first control terminal, such as a remote controller, preferably a body-sensing remote controller.
  • the reference yaw direction is the reference for the first control terminal to control the flight direction of the unmanned aerial vehicle in horizontal flight.
  • the first control terminal can send horizontal control instructions (including at least one of pitch control instructions and roll control instructions) to the unmanned aerial vehicle to control the unmanned aerial vehicle to fly horizontally, and the unmanned aerial vehicle can receive the first control terminal sending When the unmanned aerial vehicle performs horizontal flight in response to the horizontal flight control instruction, it needs to use the reference yaw direction to determine the flight direction of the horizontal flight.
  • the head-mounted display device may obtain the reference yaw direction of the aircraft head from the unmanned aerial vehicle or the first control terminal.
  • the reference yaw direction of the nose of the unmanned aerial vehicle may be stored in a storage device, which may be a local storage device of the unmanned aerial vehicle, or the storage device may be a storage device provided outside the unmanned aerial vehicle,
  • the storage device may be a storage device provided at the first control terminal, and the unmanned aerial vehicle may obtain the reference yaw direction of the nose sent by the first control terminal.
  • the reference yaw direction of the aircraft nose may be the current reference yaw direction of the UAV.
  • the current reference yaw direction of the unmanned aerial vehicle may be the reference yaw direction updated according to the first yaw control instruction of the first control terminal.
  • the current reference yaw direction of the UAV is the current reference yaw direction or the updated reference yaw direction.
  • it has no impact on the principle of displaying the logo on the head-mounted display device in the embodiment of the present application.
  • the following mainly explains the principle of taking the reference yaw direction as the current reference yaw direction as an example.
  • the current reference yaw direction of the unmanned aerial vehicle is the reference yaw direction updated according to the first yaw control instruction of the first control terminal. Similar, will not be repeated here.
  • the unmanned aerial vehicle adjusts the actual yaw direction of the imaging device according to the first yaw control instruction sent by the first control terminal and/or the second yaw control instruction sent by the second control terminal.
  • the yaw control command of the head-mounted display device can be used to control the yaw rotation of the fuselage of the unmanned aerial vehicle to adjust the actual yaw direction of the imaging device fixedly connected to the fuselage;
  • the yaw control instructions of the head-mounted display device can control the yaw rotation of the gimbal to adjust the actual yaw direction of the imaging device; or the yaw control instructions of the head-mounted display device can control the yaw of the unmanned aerial vehicle. rotation and yaw rotation of the gimbal to adjust the actual yaw direction of the imaging device.
  • the actual yaw direction of the imaging device is consistent with the above
  • the actual yaw direction of the fuselage (nose) is the same in real time or after adjustment, so the actual yaw direction of the fuselage can be used to represent the actual yaw direction of the imaging device.
  • the yaw control instructions of the head-mounted display device are used to control the unmanned aerial vehicle to adjust the actual yaw direction of the imaging device and keep the reference yaw direction of the aircraft head unchanged; to prevent remote control Control terminals such as drones will be interfered by head-mounted display devices when controlling the horizontal flight of unmanned aerial vehicles.
  • the head-mounted display device detects the movement of the user's head (such as attitude, angular velocity or angular acceleration) to generate the yaw control instruction.
  • the head-mounted display device is provided with a motion sensor, and the head-mounted display device generates the yaw control instruction according to the motion of the head-mounted display device sensed by the motion sensor.
  • a head-mounted display device acquires an image captured by an imaging device carried by an unmanned aerial vehicle, and displays the image captured by the imaging device through a display device.
  • the user can adjust the posture of the head according to the image captured by the imaging device, so that the head-mounted display device detects the movement of the user's head and generates the yaw control instruction. For example, when the user is interested in the target on the left side of the image, he can turn his head to the left, so that the head-mounted display device displays the image of the target that the user is interested in.
  • the imaging device 130 is carried directly on the fuselage 110 or carried through a pan/tilt, and is used to capture images, and the images may be pictures and/or videos.
  • the yaw rotation of the imaging device 130 and the yaw rotation of the fuselage 110 are related. Further, the imaging device 130 follows the yaw rotation of the fuselage 110, or the fuselage 110 follows the yaw rotation of the imaging device 130.
  • the head-mounted display device sends a second yaw control instruction to the unmanned aerial vehicle, and the unmanned aerial vehicle can control the yaw rotation of the fuselage according to the second yaw control instruction.
  • the method of controlling the yaw rotation of the fuselage according to the second yaw control instruction may include any of the following methods:
  • the imaging device 130 is fixedly connected to the fuselage 110 in the yaw direction. Specifically, when the actual yaw direction of the imaging device 130 needs to be adjusted, the yaw direction of the fuselage 110 can be adjusted. , so that the imaging device 130 follows the rotation of the fuselage 110 to adjust the yaw direction to obtain the required yaw direction;
  • the imaging device 130 and the fuselage 110 are movably connected in the yaw direction.
  • the unmanned aerial vehicle may include a pan/tilt 140, and the imaging device 130 is installed on the pan/tilt. 140, the pan/tilt 140 is connected to the fuselage 110.
  • the pan/tilt 140 can control the yaw rotation of the imaging device 130 to adjust the actual yaw direction of the imaging device 130.
  • the pan/tilt 140 can include a yaw motor 141, which is used to control the yaw rotation of the imaging device 130. .
  • the unmanned aerial vehicle controls the yaw and rotation of the fuselage 110, and the unmanned aerial vehicle can obtain the rotation information of the yaw and rotation of the fuselage 110 (such as yaw attitude, yaw angular velocity or angular acceleration), and perform the yaw rotation according to the The rotation information controls the yaw rotation of the pan/tilt 140 to control the yaw rotation of the imaging device 130.
  • the imaging device 130 follows the yaw rotation of the fuselage 110;
  • the imaging device 130 and the fuselage 110 are movably connected.
  • the fuselage 110 follows the yaw rotation of the imaging device 130 and the unmanned aerial vehicle rotates according to the yaw rotation.
  • the second yaw control command controls the rotation of the pan/tilt, and the pan/tilt 140 controls the yaw rotation of the imaging device 130.
  • the unmanned aerial vehicle can obtain the rotation information of the yaw rotation of the imaging device (such as yaw attitude, yaw angular velocity or angular acceleration). And control the yaw rotation of the fuselage 110 according to the rotation information (for example, control the power system to control the yaw rotation of the fuselage 110 according to the rotation information).
  • the pan/tilt 140 can control the pitch rotation and/or roll rotation of the imaging device 130 to adjust the pitch direction and/or roll rotation of the imaging device 130.
  • the pan/tilt 140 may include a pitch motor and/or a roll rotation. Or a roll motor, the pitch motor and/or roll motor are used to control the pitch rotation and/or roll rotation of the imaging device 130 .
  • the first control terminal sends a first yaw control instruction to the unmanned aerial vehicle, and the unmanned aerial vehicle can control the yaw rotation of the fuselage according to the first yaw control instruction.
  • the manner in which the unmanned aerial vehicle can control the yaw rotation of the fuselage according to the first yaw control instruction may be the same as the manner in which the unmanned aerial vehicle can control the yaw rotation of the fuselage according to the second yaw control instruction.
  • the way in which the unmanned aerial vehicle can control the yaw and rotation of the fuselage according to the first yaw control instruction may be different from the way in which the unmanned aerial vehicle can control the yaw and rotation of the fuselage according to the second yaw control instruction.
  • the reference yaw direction is related to the flight direction of the UAV
  • the actual yaw direction is related to the shooting direction of the imaging device in space, that is, the user's line of sight direction.
  • the reference and the shooting direction of the imaging device are decoupled.
  • the imaging device can shoot in more directions, such as shooting to the left or right or upward or downward; it can be used without
  • the human aircraft when flying, it can perform more types of aerial photography operations, such as flying around the shooting target while the shooting direction of the imaging device is toward the center point of the shooting target (circling the shooting lens, or called brushing), or it can fly around the shooting target. Sometimes it flies backward, sideways, etc.
  • a mark indicating the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction, so that the user's flight direction reference when controlling the UAV's horizontal flight may be consistent with the aircraft nose.
  • the control terminal remote controller
  • the mark is used to indicate the deviation between the actual yaw direction and the reference yaw direction, it can also facilitate the user to control the actual yaw direction of the imaging device.
  • it can prevent the user from not understanding the flight direction reference of the unmanned aerial vehicle when viewing images captured by the imaging device; for example, when the user controls the unmanned aerial vehicle to fly forward, in some cases the images the user sees are not shot forward in the direction of flight.
  • it is easy to become disoriented for example, the user will lose the flight direction reference of the unmanned aircraft after rotating the body left and right; by displaying the logo, the user can assist in judging the flight direction reference of the unmanned aircraft and the shooting direction of the imaging device, preventing Disorientation occurs and affects the control of the unmanned aerial vehicle.
  • the embodiment of the present application can adjust the attitude of the imaging device mounted on the unmanned aerial vehicle (such as the above-mentioned imaging device and the unmanned aerial vehicle) according to the attitude of the head-mounted display device while the unmanned aerial vehicle is flying at a predetermined speed and direction.
  • the unmanned aerial vehicle such as the above-mentioned imaging device and the unmanned aerial vehicle
  • the logo is displayed on the image or on the graphical user interface of the display device of the head-mounted display device to prevent the user from becoming disoriented after turning his head, making the user experience more realistic.
  • an identification is displayed according to the reference yaw direction and the actual yaw direction, and the identification is used to indicate the reference yaw direction.
  • the identification includes a first identification. 10.
  • the position of the first identification in the image can be determined according to the reference yaw direction and the actual yaw direction; according to the position of the first identification in the image, the position of the first identification in the image can be determined.
  • the first logo is displayed in the image.
  • OA in Figure 4A is the reference yaw direction of the nose of the UAV
  • OB is the actual yaw direction of the imaging device.
  • the heading OB can determine the position of the first logo 10 as shown in FIG. 4B in the image captured by the head-mounted display device displayed by the imaging device. Since the angle of view corresponding to the image is related to the actual yaw direction OB of the imaging device, For example, the line of sight direction when the user observes the center of the image is the same as the actual yaw direction of the imaging device. Therefore, through the positional relationship between the position of the first mark in the image and the center of the image, the reference deflection of the nose of the UAV can be set.
  • the heading is indicated to the user. That is, the first mark 10 may indicate the reference yaw direction in the space where the UAV is located.
  • the method further includes: displaying a third identifier 20 in the image, where the third identifier 20 is used to indicate the position of the center of the image.
  • the user can determine the actual yaw direction of the imaging device according to the third identification 20 , so that the user can know the actual yaw direction of the current imaging device in real time, and thereby control the unmanned aerial vehicle.
  • the horizontal position of the first mark 10 in the image is consistent with that of the image.
  • the center of the image is in the same horizontal position.
  • the first mark may be one or more of a cross, a circle, an arc, a round dot, a square dot, etc., as long as the shape can serve as a mark.
  • the position of the first logo 10 in the image is the position of the geometric center of the first logo 10 in the image.
  • the position of the center of the first logo 10 in the image may be the same as the horizontal position of the center of the image.
  • the user can be prompted that the shooting direction of the current imaging device is the same as the flight direction reference of the unmanned aerial vehicle; as shown in Figure 4A and Figure 4B, the first mark 10 is symmetrical left and right about the center of the image, and symmetrical up and down. .
  • the embodiments of the present application are mainly explained by taking the horizontal position of the first logo in the image to be the same as the horizontal position of the image center of the image. It can be understood that the vertical position of the first logo in the image is the same as the horizontal position of the image center of the image. The principle that the vertical position of the image center of the above image is the same is similar and will not be described again here.
  • determining the position of the first marker in the image based on the reference yaw direction and the actual yaw direction includes: determining the reference yaw direction OA and the actual yaw direction.
  • the orientation deviation of OB between heading directions; determine the position deviation of the first mark 10 and the center of the image according to the orientation deviation; determine the position deviation of the first mark 10 and the center of the image according to the position deviation of the first mark 10 and the center of the image.
  • a location identifying 10 in the image For example, in Figure 4C, the actual yaw direction OB1 is located at a preset angle to the left of the reference yaw direction OA.
  • the first mark 10 can indicate the reference yaw direction OA of the aircraft head, and the image center can indicate the actual yaw direction OB1 of the imaging device.
  • the position deviation between the first mark and the image center is determined by the orientation deviation,
  • the first identification may be enabled to indicate the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction.
  • the orientation deviation between OA and OB includes deviation azimuth and/or deviation angle
  • the position deviation between the first mark 10 and the center position of the image includes deviation azimuth and/or deviation distance.
  • the deviation orientation of the first logo relative to the center of the image can be determined.
  • the deviation angle in the orientation deviation of OA and OB the deviation distance between the first logo and the center of the image can be determined. .
  • the deviation direction between the reference yaw direction OA and the actual yaw direction OB is the same as the deviation direction between the first identification and the image center.
  • the first mark 10 is deflected to the right relative to the center of the image; when the reference yaw direction OA is located on the right side of the actual yaw direction OB1, When the yaw direction OA is located to the left of the actual yaw direction OB1, the first mark is shifted to the left relative to the center of the image.
  • orientation deviation between the reference yaw direction OA and the actual yaw direction OB can be determined based on the deviation of the first mark relative to the center of the image, thereby helping the user to avoid getting lost. Control the unmanned aerial vehicle.
  • the deviation angle between the reference yaw direction OA and the actual yaw direction OB is positively correlated with the deviation distance of the first identification relative to the center of the image.
  • the angle between the actual yaw direction OB and the reference yaw direction OA becomes larger (that is, the angle between OB2 and OA is greater than the angle between OB1 and OA).
  • the user can determine the deviation angle between the reference yaw direction of the aircraft head and the actual yaw direction of the imaging device based on the deviation distance of the first mark relative to the center of the image, and then adjust the user's own Control methods of unmanned aerial vehicles.
  • the first mark 10 is located outside the image. Or part of the first logo 10 is located outside the image, for example, within the graphical user interface of the display device of the head-mounted display device, but outside the displayed image.
  • the preset deviation angle threshold can be determined based on the range of yaw rotation of the pan-tilt control imaging device.
  • the actual yaw direction of the imaging device can also be adjusted by controlling the yaw rotation of the fuselage and keeping the reference yaw direction unchanged.
  • the preset deviation angle threshold such as the second deviation angle threshold, may exceed the range of the yaw rotation of the pan-tilt control imaging device.
  • the user may control the actual yaw direction of the imaging device to shift toward the reference yaw direction.
  • a third logo 20 is displayed in the image, and the third logo 20 is used to indicate that the The position of the center of the image. Help the user understand the center point of the viewing angle direction of the image at this time.
  • a fourth logo 30 may also be displayed on the image.
  • the fourth logo 30 is used to indicate the deviation direction of the first logo 10 from the image.
  • the position of the fourth identification in the image may be determined according to the offset direction between the reference yaw direction and the actual yaw direction, and the fourth identifier may be displayed according to the position in the image.
  • the fourth mark 30 is located on one side edge of the image, and the one side edge corresponds to the relative orientation of the first mark and the center of the image. Please refer to Figure 4H.
  • the first logo 10 is located outside the image to the right relative to the center of the image.
  • the fourth logo can be displayed on the right side of the image.
  • the fourth logo is an arrow pointing to the right, which can prompt the user that the The reference yaw direction is located to the right of the actual yaw direction.
  • the first logo 10 includes a first left logo 11 and a first right logo 12 .
  • the first logo 10 includes a first left logo 11 and a first right logo 12 . It can be understood that the shape of the first logo in this embodiment is different from that of the first logo in Figures 4A-4H, but the display principle is the same, and both display the first logo 10 to help the user assist in judging the level flight of the unmanned aerial vehicle. direction reference.
  • the appearance of the first left logo is different from the appearance of the first right logo, so that the user can distinguish whether it is a left logo or a right logo; as shown in Figure 5B, the first left logo 11 is a left logo. Parentheses, the second left side mark 12 is a right parenthesis, and of course is not limited to this.
  • the first left logo and the first right logo are left and right symmetrical, so that the user can determine the deviation direction and deviation distance between the symmetry line of the first left logo and the first right logo and the center of the image.
  • the first left mark 11 and the first right mark 12 in Figure 5B are respectively symmetrically located on both sides of the image center of the image, and the image center is located on the symmetry line of the first left mark 11 and the first right mark 12 .
  • the coincidence between the actual yaw direction and the reference yaw direction means that the actual yaw direction of the imaging device approximately coincides with the reference yaw direction of the aircraft head, and the orientation angle is approximately 0.
  • the first left mark and the first right mark are located at the left edge and right edge of the image respectively.
  • the influence of the first left mark and the first right mark on the image observed by the user can be reduced.
  • the position of the first identification in the image includes the position of the selected target identification in the image.
  • determining the position of the first marker in the image based on the reference yaw direction OA and the actual yaw direction OB includes: determining the reference yaw direction and the actual yaw direction OB.
  • the orientation deviation between the yaw directions, the orientation deviation includes the deviation orientation and/or the deviation angle; according to the deviation orientation in the orientation deviation, the target to be selected from the first left mark and the first right mark is The displayed target identification; determine the position of the target identification in the image according to the deviation angle in the orientation deviation.
  • both the first left mark and the second right mark are selected as the target. logo.
  • both the first left identification and the second right identification are selected as the target identification, the user can determine that the orientation deviation between the actual yaw direction and the reference yaw direction is small, as the actual yaw direction The yaw direction substantially coincides with the reference yaw direction.
  • the first left identifier is selected as the target identifier;
  • the first right side mark is selected as the target mark.
  • the first left logo 11 is selected as the target logo.
  • the position of the target identification in the image may be determined according to the deviation angle in the orientation deviation.
  • the position of the first left identifier in the image is consistent with the left edge of the image.
  • the deviation distance between them is positively related to the deviation angle between the actual yaw direction and the reference yaw direction. That is, because the actual yaw direction OB1 and the reference yaw direction OA in Figure 5C
  • the deviation angle between the actual yaw direction OB2 and the reference yaw direction OA in Figure 5E is smaller than the deviation angle between the actual yaw direction OB2 and the reference yaw direction OA.
  • the position of the first left mark 11 in the image in Figure 5D is different from the left side of the image.
  • the deviation distance between the side edges is smaller than the deviation distance between the position of the first left mark 11 in the image and the left edge of the image in FIG. 5F .
  • the deviation distance between the position of the first right identification in the image and the right edge of the image and the The deviation angle between the actual yaw direction and the reference yaw direction is positively related. This part is similar to the first left mark being selected as the target mark, and will not be described again here.
  • the user can be helped to determine the current reference yaw direction and the actual yaw direction based on the first left mark or the first right mark selected as the target mark, which is beneficial to the user's control of the unmanned aerial vehicle.
  • an identification is displayed according to the reference yaw direction and the actual yaw direction, and the identification is at least used to refer to the deviation between the actual yaw direction and the reference yaw direction, as shown in Figure
  • the identification includes a second identification 40
  • the second identification 40 includes a second reference identification 41 and a second activity identification 42 .
  • the attitude of the unmanned aerial vehicle shown in Figure 4A corresponds to the schematic diagram of the logo displayed on the head-mounted display device shown in Figure 6.
  • the attitude of the unmanned aerial vehicle shown in Figure 4C corresponds to the schematic diagram of the logo displayed on the head-mounted display device shown in Figure 7.
  • the schematic diagram showing the logo corresponds to the posture of the unmanned aerial vehicle shown in Figure 4E and the schematic diagram showing the logo displayed on the head-mounted display device shown in Figure 8 or 9.
  • the second reference mark 41 is a ruler including a scale, and the second activity mark 42 can be called a cursor; the second activity mark 42 is on the second base mark 41, also That is, the position on the scale may indicate the deviation between the actual yaw direction OB and the reference yaw direction OA.
  • the deviation between the actual yaw direction and the reference yaw direction can be indicated by the deviation angle of the second movable mark 42 relative to the second reference mark 41 .
  • the second reference mark 41 indicates the reference yaw direction OA
  • the second active mark indicates the actual yaw direction OB
  • the second active mark indicates the reference yaw direction OA
  • the second reference mark indicates the actual yaw direction OB.
  • the embodiment of the present application mainly takes the second reference mark 41 to indicate the reference yaw direction OA and the second movable mark 42 to indicate the actual yaw direction OB as an example.
  • displaying the mark according to the reference yaw direction and the actual yaw direction includes: displaying the second reference mark; and displaying the mark according to the reference yaw direction and the actual yaw direction.
  • Deviation display the second activity identifier.
  • the second reference mark 41 is displayed, that is, the second reference mark 41 is used as the deviation reference between the reference yaw direction and the actual yaw direction.
  • the deviation between the reference yaw direction OA and the actual yaw direction OB determines the positional relationship between the second activity identification 42 and the second reference identification 41 so that the user can determine the positional relationship between the second activity identification 42 and the second reference identification 41 according to the deviation between the reference yaw direction OA and the actual yaw direction OB.
  • the positional relationship between 42 and the second reference mark 41 determines the deviation between the reference yaw direction OA and the actual yaw direction OB.
  • displaying the second activity logo based on the deviation between the reference yaw direction and the actual yaw direction includes: determining the difference between the actual yaw direction and the reference yaw direction. the deviation between the actual yaw direction and the reference yaw direction; determine the deviation between the second movable mark and the second reference mark according to the direction deviation between the actual yaw direction and the reference yaw direction; according to the second movable mark The deviation from the second reference mark displays the second activity mark.
  • the orientation deviation between the actual yaw direction and the reference yaw direction includes a deviation azimuth and/or a deviation angle
  • the deviation between the second active identification and the second reference identification includes a deviation. Bearing and deviation angle or deviation distance.
  • the deviation orientation in the orientation deviation is used to determine the deviation orientation between the second active identification and the second reference identification. For example, when the second reference mark 41 indicates the reference yaw direction and the second movable mark 42 indicates the actual yaw direction, the difference between the actual yaw direction and the reference yaw direction is The deviation orientation is the same as the deviation orientation of the second movable mark 42 relative to the second reference mark 41 .
  • the deviation angle in the orientation deviation is used to determine the deviation distance between the second movable mark 42 and the second reference mark 41 .
  • the second reference mark 41 includes a ruler
  • the second movable mark 42 includes a cursor
  • the deviation angle between the actual yaw direction OB and the reference yaw direction OA is the same as the angle between the cursor and the ruler.
  • the deviation distance from the center is positively related.
  • the deviation angle between the actual yaw direction OB and the reference yaw direction OA is positively correlated with the deviation distance of the second active mark relative to the second reference mark; for example, as shown in Figure 4C If the deviation angle between the actual yaw direction OB1 and the reference yaw direction OA is less than the deviation angle between the actual yaw direction OB2 and the reference yaw direction OA in Figure 4E, then the second activity in Figure 7
  • the deviation distance of the mark 42 relative to the second reference mark 41 is smaller than the deviation distance of the second movable mark 42 relative to the second reference mark 41 in FIG. 8 .
  • the second movable mark 42 is located at the leftmost scale of the second reference mark 41, it can be determined that the deviation angle between the reference yaw direction and the actual yaw direction reaches a preset deviation angle threshold.
  • the second reference mark 41 is provided with a scale, the second reference mark 41 overlaps one side of the second movable mark 42 , and the second reference mark 41 overlaps with one side of the second movable mark 42 .
  • the two movable marks 42 point to the scale of the second reference mark 41 .
  • the center scale on the second reference mark 41 is used to indicate the reference yaw direction, and the scale pointed by the second movable mark 42 is used to indicate the actual yaw direction; or the second The central scale on the reference mark is used to indicate the actual yaw direction, and the scale pointed by the second movable mark is used to indicate the reference yaw direction.
  • the deviation angle in the orientation deviation is used to determine the deviation angle between the second movable mark 42 and the second reference mark 41 .
  • the deviation angle in the orientation deviation is indicated by the angle between the second reference mark 42 and the second movable mark 41 ; for example, the deviation angle between the actual yaw direction and the reference yaw direction It is positively related to the deviation angle of the second movable mark 42 relative to the second reference mark 41.
  • the principle of this embodiment is that the second reference mark 41 is a ruler including a scale, and the second movable mark 42 can be called a wandering code. The principles of the embodiments are similar and will not be described again here.
  • the second active identification and the second reference identification coincide.
  • the positions of the second movable mark and the second reference mark are the same.
  • the second movable mark 42 and the second reference mark 41 have the same horizontal position in the image; or please refer to Figure 9, the second movable mark 42 and the second reference mark 41 are at the same horizontal position in the image.
  • the second reference marks 41 may also overlap, that is, the included angle is 0.
  • the second reference mark 41 and the second movable mark 42 can be displayed at any position in the image, or can be displayed at the lower edge of the image and displayed in the center; of course, It is not limited to this, for example, an area outside the image in the display interface may also be displayed.
  • first identification 10 and the second identification 40 may appear at the same time, or one of them may be selected to identify the reference yaw direction and/or the deviation between the reference yaw direction and the actual yaw direction, As long as it can play the role of the implementation instructions provided by the embodiments of this application.
  • step S130 displays the identification according to the reference yaw direction and the actual yaw direction, including: displaying the identification according to the updated reference yaw direction and the actual yaw direction of the imaging device.
  • the reference yaw direction may be updated, for example, according to the first yaw control instruction of the first control terminal; or the identification may be displayed based on the current reference yaw direction and the actual yaw direction of the imaging device.
  • the reference yaw direction may be, for example, the current reference yaw direction.
  • the reference yaw direction of the nose of the UAV is OA and the actual yaw direction of the imaging device is consistent with OB.
  • the unmanned aerial vehicle When the unmanned aerial vehicle receives the first yaw control instruction from the first control terminal, it controls the fuselage of the unmanned aerial vehicle to deflect counterclockwise at an angle C1 in the yaw direction, and the actual yaw direction of the imaging device is adjusted to OB1; and according to The first yaw control instruction updates the reference yaw direction OA of the aircraft nose to obtain the updated reference yaw direction OA1; further, the UAV obtains the fuselage yaw corresponding to the first yaw control instruction.
  • the reference yaw direction is updated according to the angle C1 of the fuselage yaw rotation.
  • the updated reference yaw direction 0A1 of the aircraft head may be the yaw direction obtained by adding the angle of the yaw rotation of the fuselage to the reference yaw direction of the aircraft head.
  • the logo can be displayed based on the updated reference yaw direction OA1 and the actual yaw direction OB1 of the imaging device. The specific method of displaying the logo is similar to the above principle and will not be described again here.
  • the reference yaw direction of the UAV's nose is OA, which is consistent with the actual yaw direction OB of the imaging device.
  • the UAV receives the head-mounted display device, it generates and sends a second yaw control command.
  • the body of the UAV is controlled to deflect counterclockwise in the yaw direction by C1 angle, and the actual yaw direction of the imaging device is adjusted to OB1; the reference yaw direction of the nose is not updated and remains unchanged OA, that is, the first yaw direction.
  • the navigation control command does not change the reference yaw direction of the aircraft nose.
  • the logo can be displayed based on the current reference yaw direction OA and the actual yaw direction OB1 of the imaging device. The specific method of displaying the logo is similar to the above principle and will not be described again here.
  • the reference yaw direction of the UAV's nose is OA
  • the actual yaw direction of the UAV's imaging device is OB.
  • the navigation control command controls the fuselage of the unmanned aerial vehicle to deflect counterclockwise in the yaw direction by an angle C1
  • the actual yaw direction of the imaging device is adjusted to OB1
  • the reference deflection of the nose of the aircraft is adjusted according to the first yaw control command.
  • the heading OA is updated to obtain the updated reference yaw direction OA1; the logo can be displayed according to the updated reference yaw direction OA1 and the actual yaw direction OB1 of the imaging device.
  • the unmanned aerial vehicle when it receives and sends the second yaw control command generated by the head-mounted display device, it controls the fuselage of the unmanned aerial vehicle to deflect counterclockwise in the yaw direction by angle C2, and the actual yaw direction of the imaging device is adjusted to OB2, the base yaw direction of the aircraft nose is not updated and remains unchanged at OA1.
  • the logo can be displayed based on the current reference yaw direction OA1 and the actual yaw direction OB2 of the imaging device. The specific method of displaying the logo is similar to the above principle and will not be described again here.
  • the method further includes: displaying a fifth identifier 50 in the image, where the fifth identifier 50 is used to indicate the flight direction of the unmanned aerial vehicle.
  • the flight direction of the unmanned aerial vehicle may be the speed direction of the unmanned aerial vehicle.
  • the position of the fifth identification 50 in the image is determined according to the flight direction of the unmanned aerial vehicle, and the fifth identification is displayed according to the position of the fifth identification in the image.
  • the first control terminal may send a horizontal control instruction to the unmanned aerial vehicle to control the unmanned aerial vehicle to fly horizontally.
  • the unmanned aerial vehicle performs horizontal flight in response to the horizontal control, the horizontal direction is determined based on the reference yaw direction. The flight direction of the flight.
  • the method further includes: displaying a sixth logo 60 on the fifth logo 50, the sixth logo 60 being used to indicate the posture of the control terminal; wherein, the control terminal The attitude of is used to determine the control instructions sent by the control terminal to the unmanned flight.
  • the fifth mark 50 is a circle
  • the sixth mark 60 is a diagonal line on both sides of the circle.
  • the posture of the sixth identification on the fifth identification is determined according to the posture of the control terminal, and the sixth identification is displayed according to the posture of the sixth identification on the fifth identification.
  • the control terminal such as the first control terminal
  • the control terminal is provided with a motion sensor.
  • the motion sensor senses the movement of the first control terminal, such as the yaw attitude and/or pitch attitude of the first control terminal.
  • Corresponding control instructions are used to control the unmanned aerial vehicle to determine the flight direction of horizontal flight based on the reference yaw direction.
  • the attitude of the control terminal is related to the control instructions sent to the unmanned aerial vehicle.
  • the greater the angle of left and right tilt of the control terminal the greater the angular speed of the unmanned aerial vehicle flying left and right.
  • the greater the yaw angle of the control terminal the greater the yaw of the unmanned aerial vehicle.
  • the method further includes: predicting the flight trajectory of the unmanned aerial vehicle according to the flight direction of the unmanned aerial vehicle; and based on the predicted flight trajectory of the unmanned aerial vehicle, in the A seventh mark 70 is displayed in the image, and the seventh mark 70 is used to indicate the predicted flight trajectory of the unmanned aerial vehicle.
  • the flight trajectory includes, for example, the flight trajectory of the unmanned aerial vehicle within the next preset time period, which can assist the user in determining the flight direction of the unmanned aerial vehicle, so that the user can control the unmanned aerial vehicle according to the predicted trajectory.
  • the method further includes: displaying an eighth identifier 80 in the image, where the eighth identifier 80 is used to indicate the horizon or sea level of the environment corresponding to the image. It can help users determine the horizontal attitude of the UAV at this time.
  • the method further includes: displaying a ninth identifier 90 on the image, the ninth identifier 90 being used to indicate the actual yaw direction of the imaging device.
  • the ninth mark 90 can indicate the actual yaw direction of the imaging device in space; helping the user understand the orientation of the shooting direction of the imaging device in the world coordinate system at this time, as shown in Figure 6 , at this time, the shooting direction of the imaging device is north (N).
  • the ninth logo may be a logo in the form of a compass, but of course it is not limited to this.
  • Figure 13 is a virtual environment that a user using a head-mounted display device can experience in one embodiment, in which the first logo 10 (such as including The first left mark 11 and the first right mark 12), the first mark 10 is used to indicate the reference yaw direction of the aircraft nose and/or the deviation between the actual yaw direction of the imaging device and the reference yaw direction. .
  • the first logo 10 such as including The first left mark 11 and the first right mark 12
  • the first mark 10 is used to indicate the reference yaw direction of the aircraft nose and/or the deviation between the actual yaw direction of the imaging device and the reference yaw direction.
  • the head-mounted display device is provided with a motion sensor, and corresponding control instructions are generated according to the motion of the head-mounted display device sensed by the motion sensor, such as controlling and adjusting the actual yaw direction of the imaging device; the imaging device can be realized
  • the actual yaw direction follows the yaw direction of the user's head, making the user experience more realistic and giving the user the feeling of flying freely.
  • the first logo 10 and/or the second logo 40 are displayed to indicate the reference yaw direction of the aircraft head and/or the actual yaw direction of the imaging device and the reference yaw direction. deviation between.
  • the pan/tilt can control the pitch and rotation of the imaging device to adjust the pitch direction of the imaging device. In some embodiments, the pan/tilt can control the rolling rotation of the imaging device to adjust the rolling direction of the imaging device.
  • the head-mounted display device controls the imaging device to look down through the PTZ; and the head-mounted display device can display the tenth logo 102 on the image captured by the imaging device. , at least one of the eleventh identifier 103, the twelfth identifier 104, and the thirteenth identifier 105.
  • the tenth mark 102 is used to indicate the circular ruler of the space where the user is located, and the user can determine his or her orientation based on the tenth mark 102 .
  • the eleventh mark 103 is used to indicate the deviation between the actual yaw direction of the imaging device and the reference yaw direction, for example, indicating the deviation through a numerical value.
  • the twelfth mark 104 is used to indicate the reference yaw direction of the aircraft nose, and may be called a nose indicator; for example, when the head-mounted display device yaws to the left, the twelfth mark 104 yaws to the right; exemplary , the function of the twelfth logo 104 and the determination process of the display position are similar to that of the first logo 10. You can refer to the description of the first logo.
  • the main difference is that the first logo 10 is displayed when the user is looking straight up or pitching in a smaller range.
  • the twelfth logo 104 is displayed when the user's downward posture exceeds a certain level; by displaying the twelfth logo 104 and/or the first logo 10 according to the pitch posture of the user's head, it is convenient for the user to understand the actual yaw direction of the imaging device and the desired direction.
  • the thirteenth identifier 105 is used to indicate the center point of the space where the user is located, such as the user's location.
  • the method further includes: obtaining a first correction instruction; and in response to the first correction instruction, adjusting the reference yaw direction to an actual yaw direction of the imaging device; Adjust the position of the first logo in the image to the center position of the image or adjust the second reference logo in the logo to the position of the second active logo in the logo; and/or, Obtain a second correction instruction; and in response to the second correction instruction, adjust the actual yaw direction of the imaging device to the reference yaw direction of the aircraft head; adjust the center position of the image to the third The position of a mark in the image or the second movable mark is adjusted to the position of the second reference mark.
  • the reference yaw direction of the aircraft nose is inconsistent with the actual yaw direction of the imaging device.
  • the user may hope that the actual yaw direction of the imaging device is consistent with the reference yaw direction of the aircraft nose.
  • the correction instructions may include first correction instructions and/or second correction instructions.
  • the unmanned aerial vehicle may adjust the reference yaw direction of the nose of the unmanned aerial vehicle to the actual yaw direction of the imaging device in response to the first correction instruction.
  • the unmanned aerial vehicle obtains the second correction instruction, in response to the second correction instruction, the actual yaw direction of the imaging device of the unmanned aerial vehicle can be adjusted to the reference yaw direction.
  • the unmanned aerial vehicle responds to the first correction instruction and adjusts the reference yaw direction OA to the actual yaw direction OB of the imaging device; as shown in Figure 15B, the unmanned aerial vehicle responds to the first correction instruction.
  • the second correction instruction is to adjust the actual yaw direction OB of the imaging device to the reference yaw direction OA.
  • the reference yaw direction OA when adjusting the reference yaw direction OA to the actual yaw direction OB of the imaging device, the reference yaw direction OA can be updated, and based on the updated reference yaw direction OA and the The actual yaw direction OB displays the identification, such as updating the first identification and the second identification.
  • the first yaw direction OB can be calculated based on the reference yaw direction OA and the actual yaw direction OB of the imaging device at each time.
  • the logo and secondary logo were updated.
  • the update is, for example, real-time update.
  • the marker can be reset so that the marker accurately indicates the reference yaw direction and/or the actual yaw The deviation between the heading and the reference yaw heading.
  • the correction instruction may be generated by the first control terminal or the second control terminal. Further, the correction instruction may be generated by the user's correction operation detected by the first control terminal through the interaction device. In some cases, , the correction instruction may be generated by the user's correction operation detected by the second control terminal through the interaction device.
  • the first correction instruction is for the first control terminal to send an instruction for adjusting the reference yaw direction of the unmanned aerial vehicle
  • the second correction instruction is for the second control terminal to send an instruction for adjusting the reference yaw direction of the unmanned aerial vehicle. instruction.
  • the user can cause the first control terminal to output a first correction instruction by manipulating the first control terminal, and can cause the second control terminal to output a second correction instruction by manipulating the second control terminal.
  • the first control terminal and/or the second control terminal may be provided with a reset button or other triggering device, such as a microphone, etc.
  • a reset button or other triggering device such as a microphone, etc.
  • the reference yaw direction and the actual yaw direction of the imaging device can be reset to zero, so that the actual yaw direction of the user's imaging device coincides with the reference yaw direction, allowing normal control of flight.
  • the control method of the head-mounted display device includes: obtaining the reference yaw direction of the nose of the unmanned aerial vehicle; obtaining the actual yaw direction of the imaging device carried by the unmanned aerial vehicle; according to the reference yaw direction heading and the actual yaw direction display identification, the identification is used to indicate the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction; wherein, the reference yaw direction
  • the heading is the flight direction reference for the horizontal flight control of the unmanned aerial vehicle by the control terminal, so that the user can at least know through the displayed logo when the flight direction reference during the horizontal flight control of the unmanned aerial vehicle is different from the yaw direction of the imaging device.
  • the reference yaw direction of the unmanned aerial vehicle facilitates users to use the control terminal to accurately control the unmanned aerial vehicle, such as horizontal flight control of the unmanned aerial vehicle; when the mark is used to indicate that the actual yaw direction is consistent with the reference When there is a deviation between yaw directions, it can also facilitate the user to control the actual yaw direction of the imaging device.
  • FIG. 16 is a schematic block diagram of a head-mounted display device 400 provided by an embodiment of the present application.
  • the head-mounted display device 400 includes a display device 410, which is at least used to display images captured by an imaging device carried by an unmanned aerial vehicle.
  • the head-mounted display device 400 further includes one or more processors 401, and the one or more processors 401 work individually or jointly to perform the steps of the aforementioned control method for the head-mounted display device.
  • the head-mounted display device 400 further includes a memory 402.
  • the processor 401 and the memory 402 are connected through a bus 403, such as an I2C (Inter-integrated Circuit) bus.
  • I2C Inter-integrated Circuit
  • the processor 401 may be a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU) or a digital signal processor (Digital Signal Processor, DSP), etc.
  • MCU Micro-controller Unit
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • the memory 402 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk or a mobile hard disk, etc.
  • ROM Read-Only Memory
  • the memory 402 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk or a mobile hard disk, etc.
  • the processor 401 is used to run a computer program stored in the memory 402, and implement the steps of the aforementioned control method for a head-mounted display device when executing the computer program.
  • the processor 401 is used to run a computer program stored in the memory 402, and implement the following steps when executing the computer program:
  • an identification mark according to the reference yaw direction and the actual yaw direction, the identification being used to indicate the reference yaw direction and/or the deviation between the actual yaw direction and the reference yaw direction;
  • the reference yaw direction is a flight direction reference for the horizontal flight control of the unmanned aerial vehicle by the control terminal.
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the processor causes the processor to implement the head-mounted display provided in the above embodiments.
  • the steps of the device control method are not limited to:
  • the computer-readable storage medium may be an internal storage unit of the head-mounted display device described in any of the preceding embodiments, such as a hard disk or memory of the head-mounted display device.
  • the computer-readable storage medium may also be an external storage device of the head-mounted display device, such as a plug-in hard drive, a smart memory card (Smart Media Card, SMC), or a secure storage device equipped on the head-mounted display device.

Landscapes

  • Manipulator (AREA)
  • Forklifts And Lifting Vehicles (AREA)

Abstract

Procédé de commande pour un dispositif d'affichage monté sur la tête (400). Le procédé de commande consiste à : acquérir une orientation de lacet de référence (OA, OA1) du nez d'un véhicule aérien sans pilote (S110) ; acquérir une orientation de lacet réelle (OB, OB1, OB2, OB3) d'un appareil d'imagerie (130) (S120) ; et afficher des indicateurs (10, 11, 12, 20, 30, 40, 41, 42, 50, 60, 70, 80, 90, 102, 103, 104, 105) selon l'orientation de lacet de référence (OA, OA1) et l'orientation de lacet réelle (OB, OB1, OB2, OB3), les indicateurs (10, 11, 12, 20, 30, 40, 41, 42, 50, 60, 70, 80, 90, 102, 103, 104, 105) étant utilisés pour indiquer l'orientation de lacet de référence (OA, OA1) et/ou un écart entre l'orientation de lacet réelle (OB, OB1, OB2, OB3) et l'orientation de lacet de référence (OA, OA1) (S130), l'orientation de lacet de référence (OA, OA1) étant une référence de direction de mouvement de commande de mouvement horizontal d'une borne de commande (200, 300) sur un dispositif mobile. Le procédé de commande peut empêcher un utilisateur (101) d'être désorienté et d'influer sur le fonctionnement et la commande sur un dispositif mobile. Sont également prévus un dispositif d'affichage monté sur la tête (400), un système de commande et un support de stockage.
PCT/CN2022/102015 2022-06-28 2022-06-28 Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage Ceased WO2024000189A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202280049705.5A CN117693663A (zh) 2022-06-28 2022-06-28 控制方法、头戴式显示设备、控制系统及存储介质
PCT/CN2022/102015 WO2024000189A1 (fr) 2022-06-28 2022-06-28 Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/102015 WO2024000189A1 (fr) 2022-06-28 2022-06-28 Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage

Publications (1)

Publication Number Publication Date
WO2024000189A1 true WO2024000189A1 (fr) 2024-01-04

Family

ID=89383376

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/102015 Ceased WO2024000189A1 (fr) 2022-06-28 2022-06-28 Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage

Country Status (2)

Country Link
CN (1) CN117693663A (fr)
WO (1) WO2024000189A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120228734A (zh) * 2025-06-03 2025-07-01 工业云制造(四川)创新中心有限公司 基于mr与三维点云建模的机械臂高精度控制方法及系统

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150057844A1 (en) * 2012-03-30 2015-02-26 Parrot Method for controlling a multi-rotor rotary-wing drone, with cross wind and accelerometer bias estimation and compensation
US20180129212A1 (en) * 2016-11-09 2018-05-10 Samsung Electronics Co., Ltd. Unmanned aerial vehicle and method for photographing subject using the same
CN109074087A (zh) * 2017-12-25 2018-12-21 深圳市大疆创新科技有限公司 偏航姿态控制方法、无人机、计算机可读存储介质
CN112154480A (zh) * 2019-09-29 2020-12-29 深圳市大疆创新科技有限公司 可移动平台的定位方法、装置、可移动平台及存储介质
CN112975972A (zh) * 2021-03-01 2021-06-18 中电海康集团有限公司 一种基于头戴式ar设备和操作手柄的机器人控制方法
CN113272754A (zh) * 2020-09-28 2021-08-17 深圳市大疆创新科技有限公司 无人机的控制方法、装置、无人机、控制终端及系统
CN113994292A (zh) * 2020-05-26 2022-01-28 深圳市大疆创新科技有限公司 无人飞行器的控制方法和设备
CN114641743A (zh) * 2020-12-25 2022-06-17 深圳市大疆创新科技有限公司 无人机及其控制方法、系统、手持控制设备、头戴设备

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150057844A1 (en) * 2012-03-30 2015-02-26 Parrot Method for controlling a multi-rotor rotary-wing drone, with cross wind and accelerometer bias estimation and compensation
US20180129212A1 (en) * 2016-11-09 2018-05-10 Samsung Electronics Co., Ltd. Unmanned aerial vehicle and method for photographing subject using the same
CN109074087A (zh) * 2017-12-25 2018-12-21 深圳市大疆创新科技有限公司 偏航姿态控制方法、无人机、计算机可读存储介质
CN112154480A (zh) * 2019-09-29 2020-12-29 深圳市大疆创新科技有限公司 可移动平台的定位方法、装置、可移动平台及存储介质
CN113994292A (zh) * 2020-05-26 2022-01-28 深圳市大疆创新科技有限公司 无人飞行器的控制方法和设备
CN113272754A (zh) * 2020-09-28 2021-08-17 深圳市大疆创新科技有限公司 无人机的控制方法、装置、无人机、控制终端及系统
CN114641743A (zh) * 2020-12-25 2022-06-17 深圳市大疆创新科技有限公司 无人机及其控制方法、系统、手持控制设备、头戴设备
CN112975972A (zh) * 2021-03-01 2021-06-18 中电海康集团有限公司 一种基于头戴式ar设备和操作手柄的机器人控制方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120228734A (zh) * 2025-06-03 2025-07-01 工业云制造(四川)创新中心有限公司 基于mr与三维点云建模的机械臂高精度控制方法及系统

Also Published As

Publication number Publication date
CN117693663A (zh) 2024-03-12

Similar Documents

Publication Publication Date Title
JP7491300B2 (ja) 情報処理装置、情報処理方法、及びコンピュータが読み取り可能な記録媒体
US9886033B2 (en) System for piloting a drone in immersion
US9738382B2 (en) Drone immersion-piloting system
CN104828256B (zh) 一种智能多模式飞行拍摄设备及其飞行控制方法
CN108769531B (zh) 控制拍摄装置的拍摄角度的方法、控制装置及遥控器
JP4012749B2 (ja) 遠隔操縦システム
CN113795803B (zh) 无人飞行器的飞行辅助方法、设备、芯片、系统及介质
WO2020143677A1 (fr) Procédé de commande de vol et système de commande de vol
WO2020157955A1 (fr) Dispositif d'affichage d'objet virtuel et procédé d'affichage d'objet virtuel
WO2018184232A1 (fr) Procédé de commande à distance à détection de corps, appareil de commande, cardan et véhicule aérien sans pilote
WO2022141314A1 (fr) Procédé et appareil de planification d'itinéraire, dispositif, véhicule aérien sans pilote et support de stockage lisible
CN108780321B (zh) 用于设备姿态调整的方法、设备、系统和计算机可读存储介质
KR20170090888A (ko) Hmd를 이용한 무인기 조종 장치
JP7435599B2 (ja) 情報処理装置、情報処理方法、及びプログラム
WO2024000189A1 (fr) Procédé de commande, dispositif d'affichage monté sur la tête, système de commande et support de stockage
US20210034052A1 (en) Information processing device, instruction method for prompting information, program, and recording medium
US12456296B2 (en) Video display method and display system based on unmanned aerial vehicle viewing angle
CN108475064B (zh) 用于设备控制的方法、设备和计算机可读存储介质
KR102542181B1 (ko) 360도 vr 영상 제작을 위한 무인 비행체 제어 방법 및 장치
US20240288875A1 (en) Control method and control system for unmanned aerial vehicle
WO2022094808A1 (fr) Procédé et appareil de commande de prise de photographies, véhicule aérien sans pilote, dispositif et support de stockage lisible
CN114641746A (zh) 可移动平台及其控制方法、装置
WO2023184099A1 (fr) Procédé et appareil de commande, véhicule aérien sans pilote, système de commande et support de stockage
CN108319295A (zh) 避障控制方法、设备及计算机可读存储介质
WO2022134300A1 (fr) Procédé de guidage de vol, procédé d'étalonnage de moteur électrique, dispositif d'affichage et support de stockage lisible

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 202280049705.5

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22948309

Country of ref document: EP

Kind code of ref document: A1