RU2840814C1 - Aerial photography data processing system for determining optimum zones and controlling operating modes of target loads of unmanned aerial vehicle - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: aviation aerial survey data processing system for the unmanned aerial vehicle (UAV) target loads optimal zones determining and control over the unmanned aerial vehicle (UAV) target loads operation modes contains the target loads unit, computing device including input-output interfaces unit, aerial photography data processing unit, objects of interest detection and basic classification unit, data storage unit, data generation unit for ICS, feature vectors calculation unit, object of interest class updating unit, besides, it comprises data transceiver unit, control panel data transceiver unit, control panel data processing and analysis unit.

EFFECT: enabling increase in the target loads operation zones and their modes determination accuracy due to the aerial survey data intelligent processing performance, coordinated with the UAV operating modes, determined by the type of the solved search task.

2 cl, 1 dwg

Description

The invention relates to the field of aviation technology and can be used on unmanned aerial vehicles (UAVs) using a wide range of information processing technologies, including artificial intelligence technologies, to increase the level of autonomy and functionality of the UAV, as well as to increase the efficiency of controlling the UAV and target loads when solving a wide range of search tasks related to searching, classifying and determining the coordinates of the location, trajectory and speed of movement of objects of interest.

Modern UAVs perform a wide range of tasks, the efficiency of which largely depends on the applied flight and target load control technologies. One of the conditions for performing search tasks is the efficient control of target loads (for example, optical and multispectral cameras and radar stations) that collect information about the surrounding environment and objects of interest during aerial photography, as well as determining the optimal zones and operating modes of target loads. Determining optimal zones and controlling the operating modes of target loads during UAV flight will allow for the efficient solution of search tasks both in the UAV remote manual control mode and in the autonomous control mode with varying levels of autonomy.

An intelligent automatic control system for an unmanned aerial vehicle is known according to patent RU 164139 (IPC B64C 13/00; B64C 19/02; G05D 1/00; G01S 19/00, published on 20.08.2016), which consists of a low-level control module containing a low-level control module controller based on a 32-bit microprocessor, a neural network chip, a COM port, interfaces for connecting motion control devices, interfaces for connecting external sensors, a GNSS receiver, a power connector, a power monitoring module, four baroaltimeters, four magnetometers, four accelerometers, four gyroscopes, four thermometers, and a high-level control module containing a high-level control module controller based on a multi-core 32-bit microprocessor combined with a graphics processor, a video input, two USB ports, a Wi-Fi module and a data storage device, wherein the inputs of the neural network chip, the COM port, the interfaces for connecting motion control devices and the interfaces for connecting external sensors are connected to the outputs of the low-level control module controller, the outputs of the power supply monitoring module, the neural network chip, the COM port, the interfaces for connecting motion control devices, the interfaces for connecting external sensors, the GNSS receiver, each baroaltimeter, each magnetometer, each accelerometer, each gyroscope, each thermometer are connected to the inputs of the low-level control module controller, the video input, the outputs of the graphic processor, each USB port, the Wi-Fi module and the data storage device are connected to the inputs of the high-level control module controller, the inputs of the graphic processor, each USB port, the Wi-Fi module and the data storage device are connected to the outputs of the high-level control module controller, the inputs of the low-level control module controller are connected to the outputs of the high-level control module controller, and the outputs high-level control controller are connected to the inputs of the low-level control module controller, in addition, the high-level control module is powered from the low-level control module. The disadvantages of the system are: the inability to connect different types of target loads (TL), the inability to issue recommendations for TL control, the inability to adjust elements of search tasks during the flight, such as objects of interest.

From patent RU 195749 (IPCВ64С 13/10, В64С 19/02, G01C 23/00, published 05.02.2020) an intelligent technical vision system of an unmanned aerial vehicle is known for solving navigation problems, constructing a three-dimensional map of the surrounding space and obstacles and autonomous patrolling, which consists of a software and hardware complex consisting of an on-board computer based on a single-board computer with an expansion board based on a 32-bit controller to implement the possibility of processing signals in real time using a neural network, and an inertial measuring unit, hardware located on the expansion board; an on-board video camera for receiving a video stream and transmitting it to the on-board computer via a standard USB interface; an RGB-D camera for receiving a depth field in the field of view of the device and transmitting it to the on-board computer via a standard USB interface; a flight controller connected through the control system wiring of the unmanned aerial vehicle body to an expansion board for distributing and transmitting commands controlling the rotation speed of the engines and used to control the stabilization system of the unmanned aerial vehicle; a radio signal receiver for receiving signals from a third-party control system and transmitting them for processing to the on-board computer via the standard S.BUS serial communication interface; a rotating laser scanner-rangefinder for determining distances to objects in the environment and transmitting this information to the on-board computer via the standard UART serial communication interface; a stationary laser rangefinder for determining point distances to objects in the environment and transmitting this information to the on-board computer via a standard UART serial communication interface, with the inertial measuring unit consisting of a gyroscope that determines the angular velocity of rotation around its own axes X, Y, and Z, an accelerometer that determines the magnitude of the acceleration of gravity along the axes X, Y, and Z, a compass that determines the angles between the sensor's own axes X, Y, and Z and the lines of force of the Earth's magnetic field, a barometer that determines atmospheric pressure, altitude above sea level, and temperature, and control signals to the speed controllers are carried out using the pulse-width modulation method. The disadvantages of the system are the impossibility of determining the coordinates of the location of a single object, the lack of the ability to clarify the classes of objects of interest during the flight.

From the invention patent RU 2819590 (IPC GOlC 23/00, B64U 101/18, B64U 20/80, published 05/21/2024), an on-board intelligent search and guidance system for an unmanned aerial vehicle is known, which contains a surveillance camera interconnected by inputs and outputs via information exchange channels, a computing device including a ground object detection and recognition unit, a coordinate information generation unit, a ground object tracking and identification unit, an input-output and information exchange control unit, a database of ground object types and classes, a random access memory device for storing neural network weighting coefficients and target information, an on-board systems state monitoring unit, an angular deviation determination unit, an aircraft operating mode control unit, and a video image processing unit interacting via a computing information exchange channel. The disadvantages of the system are: the lack of the ability to connect various types of aerial photography devices, the lack of the ability to use a set of neural networks to process data from various types of CNs in various operating modes, the lack of the ability to refine the set of object classes during the flight.

The technical solution known from patent RU 2819590 is selected as the closest analogue of this invention.

The problem that this invention is aimed at solving is the creation of an aviation system for processing aerial photography data to determine optimal zones and control the operating modes of target loads of an unmanned aerial vehicle, ensuring an increase in the operating efficiency of the target payload and the level of autonomy of the UAV when solving search tasks by eliminating the indicated shortcomings.

The technical result achieved by implementing the claimed invention consists in increasing the accuracy of determining the zones of operation of target loads and their modes due to the implementation of intelligent data processing (IDP) of aerial photography, coordinated with the operating modes of the UAV, determined by the type of search task being solved.

The specified technical result is achieved in that the aviation system for processing aerial photography data for determining optimal zones and controlling the operating modes of target loads of an unmanned aerial vehicle, comprising a computing device including an input-output interface unit and an aerial photography data processing unit, interconnected by a data bus, includes a CN unit, which is configured to receive data from the information and control system (ICS) of the UAV, an interconnected data reception and transmission unit and a control point (CP) data reception and transmission unit, a CP data processing and analysis unit, the input-output of which is connected to the input-output of the CP data reception and transmission unit, wherein the input-output of the data reception and transmission unit is connected to the input-output of the input-output interface unit of the computing device, which additionally includes a detection and basic classification unit of objects of interest, a data storage unit, and a data generation unit for the ICS, interconnected by a data bus. a block for calculating feature vectors and a block for refining the class of an object of interest, the input-output of the CN block is connected to the input-output of the input-output interface block, which is designed with the possibility of transmitting data to the UAV ICS.

The technical result is also achieved by the fact that the target load block 1 includes an optical-electronic system, a digital aerial photography camera and an onboard radar station.

The essence of the claimed aviation system for processing aerial photography data for determining optimal zones and controlling the operating modes of target loads of an unmanned aerial vehicle is explained by an example of its implementation and a drawing showing the structural electrical diagram of the system.

The following designations are introduced on the drawing:

1 - target load block;

2 - computing device;

3 - data reception and transmission unit;

4 - data transmission and reception unit PU;

5 - data processing and analysis unit;

6 - input/output interface block;

7 - aerial photography data processing unit;

8 - block for detection and basic classification of objects of interest;

9 - data storage block;

10 - data generation unit for the IUS;

11 - block for calculating feature vectors;

12 - block for refining the class of the object of interest;

13 - data bus.

An aviation system for processing aerial photography data for determining optimal zones and controlling the operating modes of target loads of an unmanned aerial vehicle includes (Fig. 1): a target load unit 1, a computing device 2, a data reception and transmission unit 3, a control center data reception and transmission unit 4, and a control center data processing and analysis unit 5. The computing device 2 includes an input/output interface unit 6, an aerial photography data processing unit 7, an object of interest detection and basic classification unit 8, a data storage unit 9, a data generation unit 10 for the IMS, a feature vector calculation unit 11, an object of interest class refinement unit 12, and a data bus 13.

Block 1 of target loads is designed to collect and output aerial photography information from various types of connected target loads and is, for example, a target load composition consisting of an optical-electronic system, digital aerial photography cameras, and an onboard radar station.

Computing device 2 is intended for receiving/transmitting information from external connected devices, processing it in blocks, including IOD for the tasks of detecting and classifying objects of interest, determining their parameters and/or semantic segmentation.

Block 6 of input/output interfaces is designed to convert information during interaction of external connected devices, internal blocks and data bus and represents, for example, a set of 100/1000 BASE-TX Ethernet, FC-AV interfaces, multiplex information exchange channel.

Block 7 of aerial photography data processing is intended for processing the information received from the active CN in accordance with the requirements of the subsequent IOD and the CN operating modes. Block 6 can be implemented, for example, on the basis of the ARM Cortex A9 processor.

Block 8 for detection and basic classification of objects of interest is designed to perform detection and basic classification of objects of interest based on information received from various types of CN using the corresponding neural network models and specified classifiers, taking into account the CN operating modes. Block 8 is, for example, an ATLAS 200 neural accelerator module with loaded YOLOv51 and ResNet neural network models for performing IOD in the "Detection and Classification" mode and with loaded UNet neural network models for performing IOD in the "Semantic Segmentation" mode.

Data storage unit 9 is intended for storing data on loaded neural network models, incoming aerial photography data, IOD results, information on the CN operating modes, and other service information. Unit 9 is, for example, an SDSDQAF3/128 GB memory card.

Block 10 for generating data for the IUS is intended for generating data obtained as a result of the IOD in accordance with the specified requirements. Block 10 can be implemented, for example, on the basis of the ARM Cortex A9 processor.

Block 11 for calculating feature vectors is designed to calculate feature vectors of the object of interest entering the procedure of additional classification using loaded neural network models. Block 11 is, for example, a software implementation of the ShuffleNet-v2 neural network model, executed on the basis of the ATLAS 200 neural accelerator module.

Block 12 of class refinement is intended for determining the reference class of the object of interest entering the procedure of additional classification using a model based on the support vector method or a fully connected neural network model. Block 12 can be implemented, for example, on the basis of the ATLAS 200 neuroaccelerator module.

Block 3 of data reception and transmission is intended for processing information on the results of the IOD and the operating modes of the UAV, the CN and the IOD blocks, end-to-end transmission of data received from the active CN, control signals and requests from the PU. Block 3 can be implemented, for example, on the basis of the ARM Cortex A9 processor.

Block 4 of the PU data reception and transmission is intended for processing information on the results of the IOD and the operating modes of the UAV, the CN and the IOD blocks, end-to-end transmission of data received from the active CN, sending control signals and requests from the PU. Block 4 can be implemented, for example, on the basis of the ARM Cortex A9 processor.

Block 5 of processing and analysis of PU data is intended for preparation of data on reference classes and their corresponding feature vectors for reclassification procedures. Block 4 is, for example, a server including 2 Elbrus 8SV processors with 4 installed NMCard neuroaccelerators.

Data bus 13 is intended to provide information exchange between elements of computing device 2.

The general principle of operation consists of sequential processing of aerial photography data received from the active CN in various operating modes using the corresponding IOD modes using a preset set of neural network models, the results of which are used to generate information on detected and classified objects of interest, on the basis of which decisions are made on the selection of optimal operating zones and CN operating modes. The obtained results are sent to the UAV I&C, CN and CP. At the CP, they are checked and, if necessary, a refined classification is made, the results of which are sent on board the UAV together with additional information on newly identified reference classes of objects. The refined classes are taken into account for additional classification of objects of interest on board.

The system works as follows:

To perform UAV search tasks, the following initial parameters are set: types, modes and zones of operation of the central network, types of neural network models, classifiers of objects of interest.

When the UAV performs search tasks, according to the specified parameters, from the target load block 1, aerial photography data from the active target load, the type of which is determined by the operating mode and settings of the IOD, determined by the control unit or the UAV I&C, is transmitted to the computing device 2 through the input-output interface block 6.

From the input/output interface block 6, aerial photography data is fed via the data bus 13 to the data reception/transmission block 3 for the implementation of end-to-end data transmission via the control center data reception/transmission block 4 to the control center data processing and analysis block 5, in the event that the end-to-end data transmission mode is enabled.

From the input/output interface block 6, via the data bus 13, data is transmitted to the data storage block 9 for temporary storage and further transmission for processing to the aerial photography data processing block 7.

From the data storage unit 9, the data are transmitted via the data bus 13 to the aerial photography data processing unit 7, where they are prepared for further IOD, depending on the IOD operating mode, for example, using noise level assessment, geometric distortion correction, frame thinning. In this case, the type of processing depends on the type and operating mode of the active load used for aerial photography, as well as the search task being performed.

The prepared data in the form of frames from the data storage unit 9 are fed to the block 8 for detection and basic classification of objects of interest, in which the IOD corresponding to the current mode "Detection and classification" or "Semantic segmentation" is performed. The result of the operation of the block 8 for detection and basic classification of objects of interest is a detected object of interest, or a set of objects of interest, or a group of objects of interest defined as an area on a frame element specified by the coordinates of key points, its boundaries within the frame, as well as its base class and identifier of the base class. In this case, for each type of aerial photography and IOD operation mode, its own set of objects is formed, specified by the corresponding classifiers pre-loaded into the data storage unit 9.

The obtained objects of interest, consisting of an object identifier, an image region with the object of interest (portrait), metadata, including coordinates and a confidence coefficient, and a base class identifier, are transmitted via data bus 13 to data storage unit 9 for their long-term storage in memory.

The received object of interest, consisting of an object identifier, a portrait, metadata and a base class identifier, is transmitted to the data generation unit 10 for the IMS, where it is additionally processed, including the calculation of the geodetic coordinates of the object (WGS-89, P3-90.11), the calculation of the trajectories of the moving objects, the calculation of the speed of the moving objects, for presentation in the form of forms, which are transmitted via the data bus 13 to the input-output interface unit 6, from where they are transmitted to the data reception and transmission unit 3 for further transmission via the data reception and transmission unit 4 of the PU to the data processing and analysis unit 5 of the PU, and are also transmitted via the data bus 13 to the input-output interface unit 6 for transmission to the IMS.

Data on the object of interest is transmitted via data bus 13 from data storage unit 9 to block 11 for calculating feature vectors used to refine the base class of the object.

Data on the object of interest is transmitted via data bus 13 from data storage unit 9 to feature vector calculation unit 11, where it is subjected to IOD, the result of which is a formed set of the identifier of the base class of the object of interest, the identifier of the object of interest and the calculated feature vector.

The obtained set of the identifier of the base class of the object of interest, the identifier of the object of interest and the calculated feature vector is transmitted to the data storage unit 9 via the data bus 13.

The object of interest containing the object identifier, portrait, metadata and base class identifier is transmitted through the data reception and transmission unit 3 via the data reception and transmission unit 4 PU to the data processing and analysis unit 5 PU.

Based on portraits or frames containing data of objects of interest, a set of reference objects and their classes for further classification is formed in block 5 for processing and analyzing data on the control unit.

The obtained set of identifiers of reference objects and their refined classes is transmitted to the computing device 2 via the data reception and transmission unit 4 PU and via the data reception and transmission unit 3, the input-output interface unit 6, the data bus 13 and is transmitted to the data storage unit 9.

The feature vectors of objects obtained in the feature vector calculation block 11 and stored in the data storage block 9, as well as the set of identifiers of reference objects and their classes obtained from the control unit and stored in the data storage block 9, are transmitted via the data bus 13 to the class refinement block 12, where they are used for rapid training of the fully connected neural network.

The vector of object features obtained in the feature vector calculation block 11 is transmitted from the data storage block 9 via the data bus 13 to the class refinement block 13, where it is subjected to IOD, the result of which is a refined class of the object, determined as the result of the output of a fully connected neural network.

The refined class in the form of an identifier of the found reference class of the object of interest is transmitted via data bus 13 to data storage unit 9.

The form consisting of the object identifier, the base class identifier, the base class type, the object coordinates on the frame, the frame identifier, the refined class identifier, the refined class type, the confidence coefficient is transmitted via the data bus 13 to the data generation unit 10 for the IMS, in which the data for the UAV IMS and the CN are generated. The received data via the data bus 13 are transmitted via the input/output interface unit 6 to the IMS, which generates control signals to the active CN and the UAV control subsystem according to the algorithms specified therein, as well as via the input/output interface unit 6 to the target load unit 1.

The declared system is implemented in the product ASPM.466226.003-01 On-board information and analytical system "BIAS basic", as well as in the product ASPM.466448.001 Software and hardware complex of information support of BIAS "PAK IO BIAS". The results of preliminary tests of BIAS confirm the achievement of the declared technical result due to the implementation on board the UAV of processing of species and parametric information from a set of various types of CN and the formation of IOD results for the UAV ICS, as well as due to the interaction of the board and the control unit for the exchange of information and the implementation of additional IOD on the control unit for the implementation of additional classification on board during the flight.

The implementation of the invention into the UAV does not require significant changes to the standard on-board systems, since interaction with the standard on-board systems is ensured by connecting via standard interfaces and communication channels.

Claims (2)

1. An aircraft system for processing aerial photography data for determining optimal zones and controlling the operating modes of target payloads of an unmanned aerial vehicle, comprising a computing device including an input/output interface unit and an aerial photography data processing unit, interconnected by a data bus, characterized in that the system includes a target payload unit, which is configured to receive data from the information and control system of the unmanned aerial vehicle, interconnected data receiving and transmitting unit and a control center data receiving and transmitting unit, a control center data processing and analysis unit, the input/output of which is connected to the input/output of the control center data receiving and transmitting unit, wherein the input/output of the data receiving and transmitting unit is connected to the input/output of the input/output interface unit of the computing device, which additionally includes a detection and basic classification unit of objects of interest, a data storage unit, a data generation unit for the information and control system, a feature vector calculation unit and a unit for refining the class of an object of interest, interconnected by a data bus, the input/output of the target payload unit is connected to input-output of the input-output interface block, which is designed with the ability to transmit data to the information and control system of the unmanned aerial vehicle. 2. The system according to paragraph 1, characterized in that the target payload unit includes an optical-electronic system, a digital aerial photography camera, and an onboard radar station.

Images