RU231258U1 - ON-BOARD INFORMATION AND ANALYTICAL SYSTEM - Google Patents

Abstract

The utility model relates to methods for constructing intelligent systems based on a wide range of information processing technologies, including artificial neural networks, and can be used to increase the level of autonomy of unmanned aerial vehicles (UAVs) up to the creation of fully autonomous models. The system includes a housing, a base board with a power source, combining a processor unit including a video processing module, a large binary object storage module, a metadata storage module, a time series storage module, a control module and a data bus, a neuroaccelerator unit including a detection and classification module, a semantic segmentation module, a navigation module and a data bus, an interface unit and a data storage device. The technical result consists in expanding the functional capabilities, which together, due to the implementation of data processing directly on board the UAV, make it possible to increase the level of autonomy of the unmanned aerial vehicle. 3 fig.

Description

The utility model relates to methods for constructing intelligent systems based on a wide range of information processing technologies, including artificial neural networks, and can be used to increase the level of autonomy of unmanned aerial vehicles (UAVs) up to the creation of fully autonomous models.

The level of autonomy is becoming an increasingly important characteristic for modern UAV systems. The task of increasing the level of autonomy is especially relevant in complex signal-interference conditions, when the flight mission can only be performed partially or completely autonomously.

The following key functional capabilities can be identified that affect the level of autonomy of the UAV:

automatic detection and classification of objects of interest;

escorting objects of interest;

formation of a surface model based on semantic segmentation;

formation of recommendations for the on-board information and control system for managing the carrier and payloads;

navigation without GNSS.

The prior art discloses an intelligent automatic control system for an unmanned aerial vehicle according to patent RU 164139 (IPC B64C 13/00; B64C 19/02; G05D 1/00; G01S 19/00, published on 20.08.2016), which consists of a low-level control module containing a low-level control module controller based on a 32-bit microprocessor, a neural network chip, a COM port, interfaces for connecting motion control devices, interfaces for connecting external sensors, a GNSS receiver, a power connector, a power monitoring module, four baroaltimeters, four magnetometers, four accelerometers, four gyroscopes, four thermometers, and a high-level control module containing a high-level control module controller based on a multi-core 32-bit microprocessor combined with a graphics processor, a video input, two USB ports, a Wi-Fi module and a data storage device, wherein the inputs of the neural network chip, the COM port, the interfaces for connecting motion control devices and the interfaces for connecting external sensors are connected to the outputs of the low-level control module controller, the outputs of the power supply monitoring module, the neural network chip, the COM port, the interfaces for connecting motion control devices, the interfaces for connecting external sensors, the GNSS receiver, each baroaltimeter, each magnetometer, each accelerometer, each gyroscope, each thermometer are connected to the inputs of the low-level control module controller, the video input, the outputs of the graphic processor, each USB port, the Wi-Fi module and the data storage device are connected to the inputs of the high-level control module controller, the inputs of the graphic processor, each USB port, the Wi-Fi module and the data storage device are connected to the outputs of the high-level control module controller, the inputs of the low-level control module controller are connected to the outputs of the high-level control module controller, and the outputs high-level control controller are connected to the inputs of the low-level control module controller, in addition, the high-level control module is powered from the low-level control module. The disadvantages of the system are: low degree of autonomy of the unmanned aerial vehicle, lack of ability to connect different types of payloads (PL), lack of ability to issue recommendations for PN control, lack of ability to scale the computing power without modifying the product.

The technical solution known from patent RU 164139 was selected as the closest analogue of this utility model.

The task, which this utility model is aimed at solving, is to create an on-board information and analytical system (BIAS) for data processing in real time directly on board the UAV, ensuring the execution of various technological cycles for processing species and parametric information, including processing using neural networks, the formation of recommendations for the automatic control system (ACS) for controlling the UAV and PN, ensuring adaptation to various types of target algorithms and PN in order to increase the level of autonomy of the UAV.

The technical result consists in expanding the functional capabilities of UAV control systems, which, due to the implementation of data processing directly on board, together ensures the complete autonomy of the UAV.

The specified technical result is achieved in that in the BIAS, including a processor unit, a neuroaccelerator unit, an interface unit and a data storage device, the system additionally includes a housing and a base board with a power supply source, combining the processor unit, the neuroaccelerator unit, the interface unit and the data storage device, wherein the processor unit includes a video processing module, a large binary object storage module, a metadata storage module, a time series storage module and a control module, united by a data bus of the processor unit, to which the data storage device is connected, the neuroaccelerator unit includes a detection and classification module, a semantic segmentation module and a navigation module, united by a data bus of the neuroaccelerator unit, to which the video processing module and the control module are connected, the output of which is connected to the input of the interface unit, the interface unit is connected to the video processing module and is configured to be connected to the ground control station (GCS), the ACS and the UAV PN.

The technical result is also achieved by the fact that the processor unit, the neuroaccelerator unit, the interface unit and the data storage device are connected to the base board via unified interfaces, which allow modification of the system without the need for a complete replacement of all units.

The essence of the claimed utility model is explained by an example of its implementation and drawings, which depict:

Fig. 1 - structural electrical diagram of the BIAS;

Fig. 2 - general view of BIAS;

Fig. 3 - general view of the base board.

In Fig. 1 the following designations are used:

1 - base board;

2 - processor unit;

3 - neuroaccelerator block;

4 - interface block;

5 - data storage;

6 - video processing module;

7 - module for storing large binary objects;

8 - metadata storage module;

9 - time series storage module;

10 - control module;

11 - processor unit data bus;

12 - detection and classification module;

13 - semantic segmentation module;

14 - navigation module;

15 - data bus of the neuroaccelerator block.

BIAS solves the following tasks:

reception, integration and intelligent processing of data (IDP) from a wide range of payloads installed on UAVs, for the purpose of detecting and classifying objects in the video image from the payload, identifying the boundaries of linear and area objects and determining the current coordinates of the UAV based on the results of IDP images from the payload in the absence of satellite data;

receiving and processing telemetry from the automatic control system of the UAV, payload telemetry, as well as the results of the IOD for use in BIAS algorithms for the purpose of automatic UAV flight planning and payload control, including when performing additional reconnaissance of objects of interest;

registration and issuance to the NPU (including upon request) of data from active fire control, forms of detected objects and other results of the IOD.

The BIAS consists of a housing and a base board 1 with a power source, including a processor unit 2, a neuroaccelerator unit 3, an interface unit 4 and a data storage device 5.

The base board 1 with the power supply is the basis of the BIAS design, providing communication between the processor unit 2, the neuroaccelerator unit 3, the interface unit 4 and the data storage device 5 by means of transmitting a signal between them. The connection of the processor unit 2, the neuroaccelerator unit 3, the interface unit 4 and the data storage device 5 with the base board 1 is made by means of connectors, providing, if necessary, their replacement without changing the base board 1.

The processor unit 2 includes a video processing module 6, a large binary object storage module 7, a metadata storage module 8, a time series storage module 9 and a control module 10, connected by a processor unit data bus 11.

Processor unit 2 is based on the RK3588J processor and is designed to process incoming data, generate control commands for interfaced systems, ensure storage of various types of information with organization of access to information, indexing, searching, and presentation of information in various formats.

Video processing module 6 solves the problems of video decoding, quality control of input data (including telemetry), preliminary data processing (correction, filtering), data transcoding, data transmission to module 7 for storing large binary objects, module 8 for storing metadata, module 9 for storing time series and block 3 of the neural accelerator.

Module 7 for storing large binary objects is designed to store data received from the PN and reporting documents.

Module 8 for storing metadata is intended for storing metainformation about input data, processing results and object information.

Time series storage module 9 is designed to store telemetry and system logs.

Control module 10 solves the problems of receiving data with the results of processing (forms, frame coordinates) from block 3 of the neuroaccelerator, transmitting data to module 7 for storing large binary objects, module 8 for storing metadata and module 9 for storing time series, transmitting data through interface block 4 to the associated systems (NPU, PN and ACS of the UAV), generating control commands for the PN and ACS of the UAV, processing requests for receiving information from the associated systems (NPU).

The data bus 11 for the processor unit is intended for transmitting data between the modules of the processor unit 2, as well as the data storage device 5.

Block 3 of the neuroaccelerator includes a detection and classification module 12, a semantic segmentation module 13 and a navigation module 14, connected by a data bus 15 of the neuroaccelerator block.

The neuroaccelerator block 3 is based on the Atlas 200 AI module and is designed to process the visual and parametric information received from the video processing module 6 of the processor block 2.

The data processing architecture is based on a multi-agent distributed computing system consisting of a processor unit 2 and a neuroaccelerator unit 3, allowing flexible configuration of the data processing cycle without the need to change components. The computing system is automatically configurable, which allows for the maximum efficiency of using existing resources. In addition, the computing system allows for the creation of multiple data processing scenarios consisting of stages performed on the processor unit 2 and the neuroaccelerator unit 3. The choice of scenario occurs automatically and is determined by the currently used PN, the operating mode and a set of external conditions.

In the module 12 of detection and classification based on neural network algorithms, the IOD of the image received from the active PN is performed for the purpose of automatic detection, classification, identification, localization of objects in the image, as well as determination of the geodetic coordinates of the detected objects. Detection and classification of the following classes of objects is provided: passenger car, van, truck, bus, helicopter, airplane, military vehicle, person, boat. Objects are detected in video frames using a convolutional neural network of the YOLO family architecture, which has high accuracy and performance in real-time video processing tasks. The result of executing the detection and classification algorithms are the linear coordinates of the rectangle describing the object of a given class with the corresponding confidence coefficient. The results of executing the algorithms are transmitted to the control module 10 in the form of forms of detected objects. To exclude repetitions of the same object in the results of image processing, identification algorithms (reidentification) are performed based on extracting and comparing feature vectors from images of the detected objects. Determination of geodetic coordinates of detected and classified objects is performed in the global coordinate system (WGS-84, P3-90.11) using a terrain relief database (localization of the object on the ground).

In the semantic segmentation module 13, the IOD of the image received from the active PN is performed based on neural network algorithms for the purpose of automatic detection of linear or areal objects by forming their boundaries (contours) in the image, as well as determining the geodetic coordinates of the detected objects and the type of surface under the objects based on a three-dimensional object-oriented terrain model. Detection of linear or areal objects in video frames is performed using a convolutional neural network of the UNet family architecture, which allows for the formation of a semantic segmentation map. The algorithm for semantic segmentation of an image is a division of the image into parts (segments) with the definition of a class (label) for each pixel of the image (image classification at the pixel level, dense prediction). After semantic segmentation is performed, vectorization of the results is performed, which allows for the definition of the boundaries of a linear or areal object of a given class with the corresponding confidence coefficient. The resulting semantic segmentation map is then superimposed on the relief model to obtain an object-oriented terrain model. Based on a three-dimensional object-oriented terrain model, the geodetic coordinates of points forming the boundaries of linear and area objects, as well as the type of underlying surface at any given point within the model coverage area, are determined. The boundaries of linear and area objects and surface types are transmitted to the control module 10 in the form of forms of detected linear or area objects.

In the navigation module 14, in the absence of a signal from satellite navigation systems, the geodetic coordinates of the UAV are determined in real time based on the geographic coordinates of the center of the TV image of the underlying surface from the PN (obtained during the IOD) and the current navigation data from the ACS, the height of the relief in the center of the image, the orientation angles and the viewing angles of the PN. The solution to this problem during the IOD is performed based on the georeferencing of the TV image from the PN to the reference geodata by constructing a map of image features (descriptors) and comparing the obtained features with the feature base of the reference geodata. Satellite raster and vector coverages, as well as orthophotomaps, are used as reference geodata. To determine the coordinates of the UAV, the coordinates of the terrain section with the best coefficient of conformity to the map of TV image features are used. The result of the operation of the navigation module 14 are the coordinates of the frame corners, which are transmitted to the control module 10.

The data bus 15 of the neuroaccelerator block is intended for transmitting data between the modules of the neuroaccelerator block 3, as well as the video processing module 6 and the control module 10.

Interface block 4 provides interaction of the BIAS with the GPU, UAV control system, payload (TV camera, wide-angle TV camera, IR camera, multispectral camera). List of available interfaces MIPI CSI - 3 pcs., 1GB Ethernet - 2 pcs., USB 3.0 - 1 pc., USB 2.1 - 1 pc., UART - 2 pcs., I2C - 2 pcs., SPI - 2 pcs., GPIO - 2 pcs., PCIe 3.0 4 lane, PCIe 2.0 1 lane, eDP - 1 pc.

From the NPU, through the interface block 4, control commands and control parameters of the IOD, control commands and control parameters on request of data are transmitted to the video processing module 6 of the processor block 2. From the PN, through the interface block 4, receipts of control commands and control parameters, current parameters, video information are transmitted to the video processing module 6 of the processor block 2. From the ACS, through the interface block 4, receipts of control commands and control parameters, UAV telemetry are transmitted to the video processing module 6 of the processor block 2.

Receipts of control commands and control parameters, detection and classification results, semantic segmentation results, and UAV global localization results are transmitted to the NPU via the interface block 4 from the control module 10 of the processor unit 2. An image of an object and other data on request, compressed video information from the PN, data from the PN, and UAV telemetry are transmitted to the NPU via the interface block 4 from the video processing module 6 of the processor unit 2. Control commands and control parameters for the PN operating modes are transmitted to the PN via the interface block 4 from the control module 10 of the processor unit 2. Control commands and control parameters for the UAV, and UAV global localization results are transmitted to the ACS via the interface block 4 from the control module 10 of the processor unit 2.

Data storage 5 is designed to store information: data from payloads, reporting documents, meta-information about input data, processing results, object information, telemetry and system logs.

The case is designed to accommodate and protect the base board 1 with the source. The design of the case ensures heat dissipation from the processor unit 2 and the neuroaccelerator unit 3 to the metal base of the case.

The claimed utility model is implemented in the product ASPM.466226.003-10 On-board information and analytical system minimum BIAS-M, the general views of which are shown in Fig. 2 and Fig. 3. The results of preliminary tests of the BIAS confirm an increase in the level of autonomy of the UAV due to the processing of visual and parametric information on board the UAV and the formation of recommendations for the UAV ACS and PN.

Claims (2)

1. An on-board information and analytical system comprising a processor unit, a neuroaccelerator unit, an interface unit and a data storage device, characterized in that the system includes a housing and a base board with a power source that combines the processor unit, the neuroaccelerator unit, the interface unit and the data storage device, wherein the processor unit includes a video processing module, a large binary object storage module, a metadata storage module, a time series storage module and a control module, connected by a processor unit data bus to which the data storage device is connected, the neuroaccelerator unit includes a detection and classification module, a semantic segmentation module and a navigation module, connected by a neuroaccelerator unit data bus to which the video processing module and the control module are connected, the output of which is connected to the input of the interface unit, the interface unit is connected to the video processing module and is configured to be connected to a ground control station, an automatic control system and the payload of the unmanned aerial vehicle. 2. An on-board information and analytical system according to paragraph 1, characterized in that the processor unit, the neuroaccelerator unit, the interface unit and the data storage device are connected to the base board via unified interfaces, allowing modification of the system without the need for a complete replacement of all units.

Images