CN116820118A - A data processing method, device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the disclosure provides a data processing method, a data processing device, electronic equipment and a storage medium. The data processing method performed by the flying device may include: acquiring a video stream acquired by an image acquisition device of the flight equipment; determining flight parameters of the flight device; adding the flight parameters to supplemental enhancement information SEI of the video stream; and sending the video stream to a control device, wherein the video stream and the flight parameters are commonly used for flight control of the flight device by the control device. The flight parameters are added into the supplemental enhancement information SEI of the video stream, the video stream and the flight parameters can be transmitted to the control equipment together, the flight parameters are not easy to lose, and the efficiency and the accuracy of transmitting the video stream and the flight parameters are improved. According to the video stream and the flight parameters, the control equipment can display the video stream and the flight parameters in real time, and the efficiency of realizing real-time path planning, visual-based instant positioning, VSLAM mapping and other flight control is improved.

Description

A data processing method, device, electronic equipment and storage medium

Technical field

The present disclosure relates to the field of communication technology, and in particular, to a data processing method, device, electronic equipment and storage medium.

Background technique

With the emergence of drones and other flight equipment, there is an increasing demand for the transmission and processing of drone image video data and flight parameter data. Currently, in methods involving drone data processing, data loss and data inaccuracy are prone to occur during data processing and transmission.

Contents of the invention

In view of this, embodiments of the present disclosure disclose a data processing method, device, electronic device, and storage medium.

According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, which is executed by a flight device. The method includes:

Obtain the video stream collected by the image collector of the flight equipment;

Determine the flight parameters of the flight equipment;

Add the flight parameters to the supplemental enhancement information SEI of the video stream;

The video stream is sent to a control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device.

In one embodiment, adding the flight parameters to the supplementary added information SEI of the video stream includes: applying the flight parameters according to the collection moment of the image frame in the video and the application of the flight equipment. At this time, the flight parameters are added to the SEI corresponding to the image frame in the video stream.

In one embodiment, the flight parameters are added to the SEI corresponding to the image frame in the video stream based on the acquisition time of the image frame in the video and the application time of the flight device applying the flight parameter. including: matching the collection time of each image frame with the application time of the flight parameter; determining the position of the image frame in the video stream between the collection time and the application time of the flight parameter. ; Determine the position of the video stream to insert the SEI.

In one embodiment, determining the application moment when the flight device applies the flight parameters; adding the flight parameters to the supplementary enhancement information SEI of the video stream includes: adding the flight parameters and the The time information of the application time of the flight parameters is added to the supplementary enhancement information SEI of the video stream.

In one embodiment, the flight parameters include at least one of the following: inertial measurement unit data, wherein the inertial measurement unit data includes: acceleration, angular velocity, rotation angle; flight positioning information; flight speed; flight direction; flight altitude; Flight destination; air pressure; temperature; wind direction.

In a second aspect, embodiments of the present disclosure provide a data processing method, which is executed by a control device. The method includes:

Receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment;

The flight parameters are read from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device.

In one embodiment, the application time of the flight parameter is determined, wherein the flight parameter and the application time are jointly used for the flight control of the flight device by the control device.

In one embodiment, determining the application time of the flight parameter includes: determining the application time of the flight parameter according to the position of the SEI in the video stream, wherein the application time is the same as the application time. The SEI matches the acquisition time of the image frame at the position of the video stream.

In one embodiment, determining the application time of the flight parameter includes: determining the application time of the flight parameter based on the time information of the application time carried by the SEI.

In a third aspect, an embodiment of the present disclosure provides a data processing device, which includes:

An acquisition module, used to acquire the video stream collected by the image collector of the flight equipment;

Determination module, used to determine the flight parameters of the flight equipment;

Adding a module for adding the flight parameters to the supplementary enhancement information SEI of the video stream;

A sending module, configured to send the video stream to a control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device.

In a fourth aspect, an embodiment of the present disclosure provides a data processing device, which includes:

A receiving module, configured to receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment;

A reading module, configured to read the flight parameters from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device.

In a fifth aspect, embodiments of the present disclosure provide an electronic device. The electronic device includes: a processor and a memory for storing a computer program that can be run on the processor; when the processor runs the computer program, it executes the foregoing one or the steps of the method described in multiple technical solutions.

In a sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium that stores computer-executable instructions; after the computer-executable instructions are executed by a processor, one or more of the foregoing technical solutions can be implemented described method.

The data processing method provided by the embodiment of the present disclosure is executed by the flight equipment. The method includes: obtaining the video stream collected by the image collector of the flight equipment; determining the flight parameters of the flight equipment; adding the flight parameters to in the supplementary enhancement information SEI of the video stream; sending the video stream to the control device, where the video stream and flight parameters are jointly used by the control device to control the flight of the flight device. Here, the flight parameters are added to the supplementary enhancement information SEI of the video stream. The video stream and flight parameters can be transmitted to the control device together. The flight parameters are not easily lost, which improves the efficiency and accuracy of transmitting the video stream and flight parameters. sex. By adding flight parameters to the supplementary enhancement information SEI of the video stream and sending the video stream to the control device, the control device can obtain the video stream and flight parameter data together.

The data processing method provided by the embodiment of the present disclosure is executed by a control device. The method includes: receiving a video stream sent by a flight device and containing SEI, where the SEI includes: flight parameters of the flight device; from the The flight parameters are read from the SEI; wherein the video stream and the flight parameters are jointly used by the control device to control the flight of the flight device. Here, the control device can improve the efficiency of implementing flight control for the flight device including real-time path planning and visual-based instant positioning and map construction VSLAM based on the received and displayed video stream and flight parameters.

Description of the drawings

Figure 1 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.

FIG. 2 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.

Figure 3 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.

Figure 4 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.

Figure 5 is a schematic flowchart of a data processing method provided by an embodiment of the present disclosure.

FIG. 6 is a schematic diagram of a data processing device provided by an embodiment of the present disclosure.

FIG. 7 is a schematic diagram of a data processing device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. Those of ordinary skill in the art will not make any All other embodiments obtained under the premise of creative work belong to the scope of protection of this disclosure.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description, the terms "first\second\third" are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understandable that "first\second\third" is used in Where permitted, the specific order or sequence may be interchanged so that the disclosed embodiments described herein can be practiced in other sequences than illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the disclosure only and is not intended to limit the disclosure.

As shown in Figure 1, an embodiment of the present disclosure provides a data processing method, which is executed by a flight device. The method includes:

Step S101: Obtain the video stream collected by the image collector of the flight equipment;

Step S102: Determine the flight parameters of the flight equipment;

Step S103: Add the flight parameters to the supplementary enhancement information SEI of the video stream;

Step S104: Send the video stream to the control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device.

In one embodiment, the flying device may include a drone and/or an aerial camera.

In one embodiment, the image collector of the flight device may include a camera and/or a vision sensor, etc.

In one embodiment, the flight parameters include at least one of the following: motion parameters of the flight device. The flight parameters may specifically include: flight altitude, flight speed, flight direction, inertial measurement unit data, flight positioning information, flight time, flight equipment power, yaw data, air pressure, temperature, wind direction and/or flight destination, etc. Wherein, the inertial measurement unit (IMU, Inertial Measurement Unit) data includes: acceleration, angular velocity and/or rotation angle, etc.; the flight positioning information may include global positioning system information (GPS, Global Positioning System), etc.

In one embodiment, the supplemental enhancement information SEI (Supplemental EnhancementInformation) includes:

SEI can carry supplementary customized information in video stream transmission. The information carried by SEI can include: video encoder parameters, video copyright information, and/or camera parameters and other information. , in the embodiment of the present disclosure, the SEI is also used to carry the flight parameters.

The SEI is included in the video stream, so that the SEI is transmitted to the control device along with the video stream.

The flight control of the flight equipment by the control device mainly involves controlling the flight status and/or flight destination of the flight equipment.

Exemplarily, the flight control of the flight device by the control device includes at least one of the following:

planning the flight path of said flight equipment;

Control flight navigation of said flight equipment;

Correct the yaw phenomenon of the flight equipment;

Control the equipment operating status of the flight equipment, such as the internal and external temperatures of the flight equipment, etc.;

Set the automatic return to home and landing of the flight equipment under predetermined conditions, where the predetermined conditions may include when the electric power is less than the electric power threshold and/or when the external temperature exceeds the temperature threshold, etc.

In short, the flight parameters are carried in the SEI of the video stream and transmitted to the control device. When the control device receives the image frames in the video stream, it receives the flight parameters synchronously. Compared with sending the flight parameters separately, the loss of flight parameters can be reduced. The phenomenon. The flight parameters and the image frames in the video stream are used together by the control platform to determine the flight control of the flight equipment.

In one embodiment, steps S101 and S102 may include periodically or in real time acquiring the video stream collected by the image collector of the flight equipment and determining the flight parameters of the flight equipment.

In this way, the video stream and flight parameters are acquired periodically or in real time, and the video stream and flight parameters can be sent to the control device in time. Compared with the method without periodically acquiring the video stream and flight parameters, the video stream and flight parameters can be determined in a more timely manner. , thereby transmitting video streams and flight parameters to the control device in a timely manner.

In one embodiment, the flight parameters can be obtained through an inertial measurement unit (IMU), a GPS positioning unit, an air pressure sensor, a real-time differential positioning measurement (RTK, Real-time kinematic) unit, a ranging sensor, and/or Temperature and humidity sensors and other equipment modules are collected and detected. The IMU may be a device that measures the object's three-axis attitude angle (or angular rate) and acceleration, and the IMU data may include: acceleration, angular velocity, and/or rotation angle, etc.

In one embodiment, the flight parameters may be obtained through a processing center in the flight equipment. For example, the relative flight altitude among the flight altitudes can be calculated and obtained by the processing center based on the set predetermined altitude and the current flight altitude.

In one embodiment, the video stream in step S103 may include encoding the video stream collected by the image collector of the flight equipment.

In one embodiment, the encoding may be encoding according to a video codec standard to obtain a video code stream in a standard format.

Exemplarily, the video codec standards may include: H.264 (also known as H.264/AVC), H.265 (also known as H.265/HEVC) and H.266 (also known as H.266/ VVC) and other standards. The system framework of the video codec standard may include: video coding layer (VCL, Video Coding Layer) and network abstraction layer (NAL, Network Abstraction Layer). Wherein, the supplementary enhancement information SEI is in the network abstraction level.

In one embodiment, encoding the video stream collected by the image collector of the flight equipment and obtaining the video stream in a video codec standard format may include: encoding the video stream collected by the image collector of the flight equipment according to a streaming tool. The collected video stream is encoded to obtain a video stream in a video codec standard format, where the streaming media tool may include FFmpeg or Gstreamer, etc.

Exemplarily, encoding the video stream collected by the image collector of the flight equipment to obtain a video stream in a video codec standard format may include encoding the collected video stream in MPEG-4 format to obtain a standard H .264 video stream.

In one embodiment, the flight parameters in step S103 may include: flight parameters obtained by encoding the collected flight parameters.

In one embodiment, encoding the flight parameters includes: encoding the flight parameters in hexadecimal, binary, 64 and/or American Standard Code for Information Interchange (ASCII, American Standard Code). for Information Interchange) encoding, etc.

In one embodiment, the step S103 may further include: determining the position of the SEI in the network abstraction level at the position of the video stream; and adding the flight parameters to the supplementary enhancement information SEI of the video stream.

In one embodiment, the step S103 may include adding the flight parameters to the supplementary enhancement information SEI of the video stream when encoding the video stream.

In one embodiment, step S103 may include adding the flight parameters to supplementary enhancement information SEI in the encoded video stream. In one embodiment, the SEI is in the network abstraction level of the video codec standard, and the alternative location of the SEI in the network abstraction level may be included between the picture parameter set (PPS) and the key frame (IDR) .

In one embodiment, sending the video stream to the control device in step S104 may include sending the video stream to the control device according to a data communication network protocol, wherein the data communication network protocol may include: real-time message transmission protocol (RTMP, Real Time Messaging Protocol), Real Time Streaming Protocol (RTSP, Real TimeStreaming Protocol), Real-time Transport Protocol (RTP, Real-time Transport Protocol) and/or Hyper Text Transfer Protocol (HTTP, Hyper Text Transfer Protocol), etc.

An embodiment of the present disclosure provides a data processing method, which adds the flight parameters to the supplementary information SEI of the video stream, including:

According to the collection time of the image frame in the video and the application time of the flight device applying the flight parameter, the flight parameter is added to the SEI corresponding to the image frame in the video stream.

In one embodiment, the collection time and the application time may be determined based on a timestamp. In one embodiment, the collected video stream, the timestamp of the acquisition moment of the image frame in the video stream, the flight parameters, and the timestamp of the application moment of the flight parameters can be obtained according to the flight equipment.

In one embodiment, the acquisition time of the image frame can be determined according to the time of the image acquisition device, the application time of the flight parameters can be determined according to the time of the flight parameter determination device, and the image acquisition device and the flight parameter determination device The devices take the same time.

In one embodiment, the application time may be equal to the collection time of the collection device that collects the flight parameters.

In one embodiment, flight parameters within the same time period can be added to the SEI corresponding to the image frame in the video stream according to the acquisition time of the image frame and the flight parameter application time in the video stream.

For example, the application time and the collection time are matched, and the flight parameters corresponding to the application time are added to the one or more SEIs closest to the image frames equal to the collection time of the application time.

In one embodiment, the flight parameters may include encoded flight parameters, and the video stream may include an encoded video stream in a standard encoding format.

As shown in Figure 2, an embodiment of the present disclosure provides a data processing method, which adds the flight parameters to the flight parameters according to the collection time of the image frame in the video and the application time of the flight equipment. The SEI corresponding to the image frame in the video stream includes:

Step S201: Match the acquisition time of each image frame with the application time of the flight parameters;

Step S202: Determine the position of the image frame in the video stream at the acquisition time and the application time of the flight parameter;

Step S203: Determine the position of the video stream and insert the SEI.

In one embodiment, the step S201 may include: matching based on the same acquisition time of each image frame and the application time point of the flight parameter, and obtaining image frames and flight parameters corresponding to each other at the same time.

In one embodiment, step S201 may include: when there is no image frame at the same time point as the application time of the flight parameter, select the image frame at the acquisition time closest to the application time of the flight parameter to match the image frame .

In one embodiment, the step S201 may include: the acquisition time of one image frame corresponds to the same flight parameters at multiple application times. The multiple flight parameters may be added together to the SEI corresponding to the position of the video stream where the image frame corresponding to the time is located.

In one embodiment, the step S203 may include determining the position of the video stream and inserting the flight parameters into the SEI when encoding the video stream.

Exemplarily, when encoding a video stream, determine the position of the image frame at the collection moment and the application moment in the video stream, insert SEI at the position, and add the flight parameters at the corresponding moment. Added to SEI.

In one embodiment, step S203 may include determining the position of the video stream in the encoded video stream and inserting the flight parameter into the SEI.

Exemplarily, match the application time of the flight parameters with the collection time of each image frame in the encoded video stream; determine the image frame at the corresponding time and the position of the image frame in the video stream; The SEI is inserted into the network abstraction level corresponding to the location; the flight parameters are added to the SEI.

As shown in Figure 3, an embodiment of the present disclosure provides a data processing method, which further includes:

Step S301: Determine the application time when the flight equipment applies the flight parameters;

Step S302: Adding the flight parameters to the supplementary enhancement information SEI of the video stream includes:

Step S303: Add the flight parameters and the time information of the application time of the flight parameters to the supplementary enhancement information SEI of the video stream.

In one embodiment, the SEI is in the network abstraction level of the video codec standard. The position of the SEI in the network abstraction level may be included between the picture parameter set (PPS) and the key frame (IDR).

In one embodiment, step S302 may include adding the flight parameters and the time information of the application time of the flight parameters to the end of the identification field in the SEI. Wherein, the flight parameters and the application time of the flight parameters may adopt the same encoding method. For example, the identification field of the SEI may be: 00 00 00 01 06 field.

If the application time and flight parameters are synchronized and carried in the SEI, the control device will not only receive the flight parameters, but also accurately know the time information when the flight device uses the flight parameters. This will help the control device to better use the flight parameters. Flight navigation and other flight controls of flight equipment.

An embodiment of the present disclosure provides a data processing method. The flight parameters include at least one of the following:

Inertial measurement unit data, wherein the inertial measurement unit data includes: acceleration, angular velocity, rotation angle; flight positioning information; flight speed; flight direction; flight altitude; flight destination; air pressure; temperature; wind direction.

In one embodiment, the flight positioning information may include various satellite positioning information or positioning information based on speed sensor measurements. The satellite positioning information may include at least one of the following: positioning information obtained based on Global Positioning System positioning, and/or positioning information obtained based on Beidou Positioning System positioning.

In one embodiment, the flight altitude may include: relative takeoff altitude, fusion altitude, GPS altitude, RTK (real-time differential positioning measurement) altitude and/or altitude above the ground, etc. The relative take-off height may include the height change of the UAV relative to the take-off point; the fusion height may include the altitude using the barometer value as the initial value, using a fusion algorithm and using the data output of all sensors on the UAV. value; the GPS altitude may include the real-time altitude based on the GPS satellite output; the RTK altitude may include the real-time altitude based on the RTK satellite output; the ground altitude may include the UAV obtained with the aid of a ranging sensor on the UAV The distance to the object currently below.

As shown in Figure 4, an embodiment of the present disclosure provides a data processing method, which is executed by a control device. The method includes:

Step S401: Receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment;

Step S402: Read the flight parameters from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device.

In one embodiment, the control device may include a cloud processor, a cloud management platform, a cloud server and/or a mobile application, etc. Wherein, the control device may include a control device controlled by a user or an automatic control device with computing processing capabilities.

In one embodiment, the control device may include a control device that controls one respective flying device or a control device that controls a plurality of flying devices. The control device may include one or more servers or one or more computers that control the platform.

In one embodiment, the control device may receive the video stream containing the SEI. The captured video stream and the flight parameters transmitted together with the video stream time can be obtained according to the video stream containing the SEI. The control device can control the flight device according to the flight parameters of the video stream and the video stream time, which improves the accuracy and timeliness of the flight control of the flight device by the control device.

In one embodiment, the control device can perform real-time vision-based instant positioning and mapping (VSLAM, Visual Simultaneous Localization and Mapping) of the flight device based on the received video stream and flight parameters. For example, a map of the flight environment can be constructed in real time based on images in the video stream, and the instant positioning of the flight equipment in the current map can be determined based on information such as flight positioning information and flight altitude information in the flight parameters.

In one embodiment, the control device can perform flight control on the flight device according to the received video stream and flight parameters, which mainly involves controlling the flight status and/or flight destination of the flight device.

In one embodiment, the flight control of the flight device by the control device according to the received video stream and flight parameters includes at least one of the following:

planning the flight path of said flight equipment;

Control flight navigation of said flight equipment;

Correct the yaw phenomenon of the flight equipment;

Control the equipment operating status of the flight equipment, such as the internal and external temperatures of the flight equipment, etc.;

Set the automatic return to home and landing of the flight equipment under predetermined conditions, wherein the predetermined conditions may include when the electric power is less than the electric power threshold, when the external temperature exceeds the temperature threshold, etc.

In one embodiment, the control device planning the flight path of the flight device according to the received video stream and flight parameters may include:

Plan the flight path based on the real-time flight positioning information and flight destination information in the flight parameters and the real-time environment image information determined based on the video stream; and/or,

According to the current real-time environment information, real-time flight positioning information, real-time flight altitude, real-time flight direction and other information determined in the flight parameters and video stream, automatic obstacle avoidance flight paths are planned based on the real-time information.

In one embodiment, the control of flight navigation of the flight equipment may include: real-time calculation with VSLAM based on flight speed, flight direction, flight positioning information, acceleration in inertial measurement unit data, and/or flight destination and other flight parameters. Adjustment of flight parameters such as the path already flown, the path to the flight destination, the flight time required to the flight destination, and flight speed or acceleration.

In one embodiment, step S402 may include: determining the position of the SEI in the video stream; and reading flight parameters according to the position of the SEI.

In one embodiment, the method may include: determining the location of the SEI according to the identification field of the SEI; and reading the flight parameters according to the location of the SEI.

In one embodiment, reading the flight parameters may include: directly obtaining the flight parameters and decoding the encoded flight parameters to obtain the flight parameters.

In one embodiment, the method further includes: decoding according to the received video stream to obtain video image data. Wherein, the decoding corresponds to the encoding of the video stream by the flight equipment.

In one embodiment, the method further includes reading the flight parameters from the SEI and decoding the encoded video stream through a streaming tool, which may include FFmpeg or Gstreamer, etc. In one embodiment, reading the flight parameters from the SEI may include reading the flight parameters from the SEI through a packet capture analysis tool.

In one embodiment, the control device performs real-time data display based on the video stream and flight parameters.

An embodiment of the present disclosure provides a data processing method, which further includes:

The application time of the flight parameter is determined, wherein the flight parameter and the application time are jointly used for the flight control of the flight device by the control device.

In one embodiment, the control device can more accurately control the flight device according to the flight parameters and the application time.

An embodiment of the present disclosure provides a data processing method. Determining the application time of the flight parameters includes:

The application time of the flight parameter is determined according to the position of the SEI in the video stream, where the application time matches the acquisition time of the image frame of the SEI at the position of the video stream.

In one embodiment, this may include determining the position of the SEI in the video stream according to the SEI identification field and determining the application moment of the flight parameter.

An embodiment of the present disclosure provides a data processing method. Determining the application time of the flight parameters includes:

The application time of the flight parameters is determined based on the time information of the application time carried by the SEI.

In one embodiment, the application time of the flight parameter may be determined based on the time information of the application time carried by the SEI added by the flight device. In one embodiment, as shown in Figure 4, a data processing method may include:

Step 1. Data acquisition:

Obtain the video stream collected by the image collector of the flight device and the flight parameters of the flight device according to the flight device, wherein the acquisition time of the image frames in the video stream is the same as the application time of the flight device applying the flight parameters;

Step 2. Data processing:

Encode the video stream according to the flight equipment to obtain a standard H.264 video code stream, and locate the location of the supplementary enhancement information SEI at the H.264 network abstraction level;

Step 3. Data fusion:

Add the flight parameters to the supplementary enhancement information SEI of the video stream according to the flight equipment;

Step 4. Data transmission and reception:

The flight device sends the video stream containing flight parameters to the control device, and the control device receives the video stream and decodes the video stream and flight parameters collected by the flight device to obtain time synchronization.

In one embodiment, as shown in Figure 5, a data processing method may include:

Step S1: Obtain the video stream collected by the image collector of the flight equipment according to the flight equipment;

Step S2: Determine the flight parameters of the flight equipment according to the flight equipment, wherein the acquisition time of the image frames in the video stream is the same as the application time of the flight equipment applying the flight parameters;

Step S3: Encode the video stream collected by the image collector of the flight equipment according to the flight equipment to obtain a video code stream in a standard format, where the video code stream in the standard format may include a standard H.264 code stream. ;

Step S4: Determine the position of the supplementary enhancement information SEI in the standard format video code stream according to the flight equipment, wherein the alternative position of the SEI in the network abstraction level may be included in the picture parameter set (PPS) and key Between frames (IDR);

Step S5: Encode the flight parameters according to the flight equipment to obtain encoded flight parameters;

Step S6: Add the encoded flight parameters to the supplementary enhancement information SEI of the video stream according to the flight equipment;

Step S7: Send the video stream to the control device according to the flight device, where the video stream and flight parameters are jointly used by the control device to control the flight of the flight device;

Step S8: Receive the video stream according to the control device;

Step S9: Read the flight parameters from the SEI of the video stream according to the control device, where the reading includes determining the SEI position and reading the flight parameters;

Step S10: Decode the flight parameters and the video stream according to the control device to obtain the video stream collected by the flight device.

Step S11: The control device performs flight control on the flight device according to the decoded video stream and flight parameters, where the flight control includes planning a path and real-time vision-based instant positioning and map construction VSLAM.

For example, the step S1 may include obtaining the video stream collected by the camera of the drone and the timestamp corresponding to the collection moment of the image frame in the video stream according to the drone.

The step S2 may include obtaining the inertial measurement unit IMU data and global positioning system GPS information from the flight parameters of the drone according to the drone, and a timestamp corresponding to the application time of the flight parameters.

The step S3 may include encoding the collected video stream according to the streaming media tool FFmpeg or the streaming media tool Gstreamer in the drone to obtain a standard H.264 video stream.

The step S4 may include adding SEI between the image parameter set (PPS) and the key frame (IDR) at the network abstraction level corresponding to the H.264 code stream according to the drone. The adding SEI may include adding the SEI identification field 00 00 0001 06. Wherein, the number of images between two key frames in the H.264 code stream is fixed.

The step S5 may include hexadecimal encoding of the IMU and GPS information according to the drone.

The step S6 may include adding the encoded inertial measurement unit IMU data and GPS information to the back of the SEI identification field 00 00 00 01 06 of the video stream according to the drone.

The step S7 may include sending the H.264 code stream containing the SEI to the receiving end of the control device according to the drone through a streaming tool such as FFmpeg or Gstreamer and using a transmission protocol such as RTMP or RTSP.

The step S8 may include receiving the H.264 code stream containing the SEI according to the control device.

The step S9 may include extracting data between the SEI identification field and the next network abstraction layer header information according to the position of the SEI in the network abstraction layer of the received H.264 code stream by the control device. The data That is, flight parameters.

The step S10 may include decoding the extracted flight parameters in hexadecimal according to the control device and decoding the H.264 code stream according to the streaming tool of the control device; obtaining a time-synchronized video Flow and flight parameters.

The step S11 may include the control device performing real-time flight video and flight parameter display, planned path and/or real-time VSLAM on the flight device according to the decoded video stream and flight parameters.

Wherein, the data processing method can realize real-time synchronous transmission of the video stream and flight parameters of the flight equipment by adding the flight parameters of the flight equipment to the SEI of the video stream, which extends the supplementary enhancement in the standard H.264 code stream. The application scenarios of the information improve the synchronization and accuracy of the flight equipment video stream and flight parameters during the transmission process, and meet the needs of the control equipment for real-time data display, real-time VSLAM and path planning.

As shown in Figure 6, an embodiment of the present disclosure provides a data processing device, which includes:

The acquisition module 10 is used to acquire the video stream collected by the image collector of the flight equipment;

Determination module 20, used to determine the flight parameters of the flight equipment;

Adding module 30, configured to add the flight parameters to the supplementary enhancement information SEI of the video stream;

The sending module 40 is configured to send the video stream to a control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device.

In one embodiment, the adding module 30 is also configured to add the flight parameters to the video stream according to the collection time of the image frame in the video stream and the application time of the flight device applying the flight parameters. in the SEI corresponding to the image frame.

In one embodiment, the device further includes a matching module 31, which is used to match the acquisition time of each image frame with the application time of the flight parameters;

The determination module 20 is also used to determine the position of the image frame in the video stream at the acquisition time and the application time of the flight parameter; determine the position of the video stream and insert the SEI.

In one embodiment, the determining module 20 is also used to determine the application moment when the flight equipment applies the flight parameters; the adding module 30 is also used to combine the flight parameters and the application of the flight parameters. The time information of the time is added to the supplementary enhancement information SEI of the video stream.

In one embodiment, the flight parameters include at least one of the following:

Inertial measurement unit data, wherein the inertial measurement unit data includes: acceleration, angular velocity, and rotation angle;

Flight positioning information;

flight speed;

flight direction;

flight altitude;

flight destination;

air pressure;

temperature;

wind direction.

As shown in Figure 7, an embodiment of the present disclosure provides a data processing device, which includes:

The receiving module 50 is configured to receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment;

The reading module 60 is configured to read the flight parameters from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device.

In one embodiment, the device further includes a determination module 70, which is used to determine the application time of the flight parameters, wherein the flight parameters and the application time are jointly used for the control device Flight control of said flight equipment.

In one embodiment, the determination module 70 is further configured to determine the application time of the flight parameter according to the position of the SEI in the video stream, where the application time is consistent with the location of the SEI. The acquisition time of the image frame at the position of the video stream matches.

In one embodiment, the determination module 70 is further configured to determine the application time of the flight parameters based on the time information of the application time carried by the SEI.

It should be noted that those skilled in the art can understand that the methods provided in the embodiments of the present disclosure can be executed alone or together with some methods in the embodiments of the present disclosure or some methods in related technologies.

An embodiment of the present disclosure also provides an electronic device. The electronic device includes: a processor and a memory for storing a computer program that can be run on the processor. When the processor runs the computer program, it executes one or more of the foregoing The steps of the method described in the technical solution.

Embodiments of the present disclosure also provide a computer-readable storage medium that stores computer-executable instructions. After the computer-executable instructions are executed by the processor, the method described in one or more of the foregoing technical solutions can be implemented. .

The computer storage medium provided in this embodiment may be a non-transient storage medium.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, It includes the steps of the above method embodiment; and the aforementioned storage media includes: mobile storage devices, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc. A medium on which program code can be stored.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure. should be covered by the protection scope of this disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (13)

1. A data processing method, characterized in that it is executed by flight equipment, and the method includes: Obtain the video stream collected by the image collector of the flight equipment; Determine the flight parameters of the flight equipment; Add the flight parameters to the supplemental enhancement information SEI of the video stream; The video stream is sent to a control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device. 2. The data processing method according to claim 1, wherein adding the flight parameters to the supplementary information SEI of the video stream includes: According to the collection time of the image frame in the video and the application time of the flight device applying the flight parameter, the flight parameter is added to the SEI corresponding to the image frame in the video stream. 3. The data processing method according to claim 2, wherein the flight parameters are added to the flight parameters according to the collection time of image frames in the video and the application time of the flight equipment applying the flight parameters. The SEI corresponding to the image frame in the video stream includes: Match the acquisition time of each image frame with the application time of the flight parameters; Determine the position of the image frame in the video stream at the acquisition time and the application time of the flight parameter; The location of the video stream is determined to insert the SEI. 4. The data processing method according to claim 1, characterized in that the method further includes: Determine the application time when the flight equipment applies the flight parameters; Adding the flight parameters to the supplementary enhancement information SEI of the video stream includes: The flight parameters and the time information of the application time of the flight parameters are added to the supplementary enhancement information SEI of the video stream. 5. The data processing method according to claim 1, characterized in that the flight parameters include at least one of the following: Inertial measurement unit data, wherein the inertial measurement unit data includes: acceleration, angular velocity, and rotation angle; Flight positioning information; flight speed; flight direction; flight altitude; flight destination; air pressure; temperature; wind direction. 6. A data processing method, characterized in that it is executed by a control device, and the method includes: Receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment; The flight parameters are read from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device. 7. The data processing method according to claim 6, characterized in that the method further includes: The application time of the flight parameter is determined, wherein the flight parameter and the application time are jointly used for the flight control of the flight device by the control device. 8. The data processing method according to claim 7, wherein determining the application time of the flight parameters includes: The application time of the flight parameter is determined according to the position of the SEI in the video stream, where the application time matches the acquisition time of the image frame of the SEI at the position of the video stream. 9. The data processing method according to claim 7, wherein determining the application time of the flight parameters includes: The application time of the flight parameters is determined based on the time information of the application time carried by the SEI. 10. A data processing device, characterized in that the device includes: An acquisition module, used to acquire the video stream collected by the image collector of the flight equipment; Determination module, used to determine the flight parameters of the flight equipment; Adding a module for adding the flight parameters to the supplementary enhancement information SEI of the video stream; A sending module, configured to send the video stream to a control device, where the video stream and flight parameters are jointly used for flight control of the flight device by the control device. 11. A data processing device, characterized in that the device includes: A receiving module, configured to receive a video stream sent by the flight equipment and containing SEI, where the SEI includes: flight parameters of the flight equipment; A reading module, configured to read the flight parameters from the SEI; wherein the video stream and the flight parameters are jointly used for the flight control of the flight device by the control device. 12. An electronic device, characterized in that the electronic device includes: a processor and a memory for storing a computer program capable of running on the processor, wherein when the processor runs the computer program, claim 1 is executed to any one of the 9 data processing methods. 13. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the computer-readable storage medium can realize any one of claims 1 to 9 The data processing method described in the item.