RU2841111C1 - Method of controlling on-board systems of unmanned vehicles using neural networks based on architecture of transformers - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method of controlling on-board systems of unmanned vehicles using neural networks based on a transformer architecture. Method includes combined processing of output signals of unmanned vehicle sensors. Neural network is trained directly on the input signals of the sensors and outputs the required control signals based on the studied patterns. Architecture of the convolutional neural network directly maps input raw image pixels to control commands. Synthesis network is the basis of the model architecture and is responsible for extraction of environmental characteristics. Path prediction network receives the feature vectors provided by the synthesis network and predicts the next multiple route points for the vehicle. Synthesis network is divided into an image branch and a bird's-eye view branch. Characteristic maps from both branches are transmitted to the input of the transformer module, which performs combined processing of modality branch data.

EFFECT: improving the accuracy and performance of on-board systems of unmanned vehicles.

2 cl, 4 dwg

Description

AREA OF TECHNOLOGY

The claimed invention relates to computer-implemented methods and systems for controlling on-board systems of unmanned vehicles using neural networks based on transformer architecture.

PRIOR ART

The field of unmanned vehicles has recently attracted much research interest due to its proximity to the implementation of fully autonomous vehicles. Data from various sensors is combined together to create a unified view of the surrounding space.

Neural network models based on the transformer architecture are used in various areas of high technology, for example, in on-board systems of unmanned vehicles. At the same time, the issue of applying the end-to-end training method of neural networks based on the transformer architecture for use in on-board systems of unmanned vehicles remains insufficiently studied.

Similar approaches are known in the art, for example, an automotive display system (see US10300856B2, published 05/28/2019), which includes a single video display screen and a video processor designed to process captured video image data captured by a camera in a vehicle. A single video display screen includes a left display area on the left side, a right display area on the right side, and a middle display area between the left and right display areas. The display screen can use each display area to display video images obtained from a corresponding portion of the captured image data. When the display screen displays video images in the right or left display area obtained from a corresponding portion of the captured image data, video images representing another portion of the captured image data on the other display are not displayed on one display screen. area, edge.

The tracking system presented in the analogue does not fully reveal the approach to reading camera data, so it is difficult to judge the object tracking system used.

Also known from the prior art are methods and systems for automated detection of the presence of objects (see RU2744012, published 02.03.2021). The method comprises receiving sensor data representing the surrounding area of the SDC in the form of three-dimensional point cloud data, generating, by means of an MLA, based on the three-dimensional point cloud data, a set of feature vectors representing the surrounding area, generating, by means of an MLA, a grid representation of the surrounding area, wherein each given cell of the grid representation includes a predicted distance parameter indicating the distance from the given cell to the nearest cell with an obstacle, and using, by means of an electronic device, the distance parameter in order to determine the presence of an obstacle in the surrounding area of the SDC.

The specified method does not have high accuracy, since it is designed only for indoor/warehouse use, and is not capable of operating in rain or snow conditions. Also, this method of determining the distance to objects does not take into account the history of the movement of the detected obstacle, information about which can significantly improve the accuracy of determining the distance to dynamic objects.

The prior art discloses methods and systems for automated detection of the presence of objects, selected as the closest analogue (see RU2769921, published 08.04.2022). The method is implemented by means of an electronic device connected with the possibility of communication with a lidar installed on a vehicle, wherein the lidar has a plurality of lasers for capturing lidar point cloud data. The method comprises: receiving first lidar point cloud data captured by the lidar; executing by the electronic device a machine learning algorithm (MLA) for the following: analyzing a first plurality of lidar points from the first point cloud data relative to a response template of the plurality of lasers; extracting data of a grid representation of the surrounding area of the vehicle; determining that the first plurality of lidar points is associated with a blind spot, wherein the blind spot prevents the detection algorithm of the electronic device from detecting the presence of at least one object in the surroundings of the vehicle, which is due to the fact that at least one object is present.

This method also does not have high accuracy, since it uses only a lidar as a sensor, which has a large amount of noise when operating in conditions of heavy rain, fog or snow. Using several sensors with data merging from them allows you to level out the shortcomings of each sensor and increase the accuracy of recognition.

The proposed analogue comes closest to solving the problems solved by the claimed invention, however, in the opinion of the applicant, the problem is not solved in the most effective way, since the method also does not ensure reliability.

SUMMARY OF THE INVENTION

This invention is aimed at solving a technical problem associated with the creation of a method for controlling on-board systems of unmanned vehicles using neural networks based on the architecture of transformers with increased accuracy and performance of controlling unmanned vehicles.

The technical result of the invention is to increase the accuracy and performance of the on-board systems of unmanned vehicles.

The technical result is achieved by creating a method for controlling on-board systems of unmanned vehicles using neural networks based on the architecture of transformers, including joint processing of output signals of sensors of the unmanned vehicle, including various types.

The method of controlling the on-board systems of unmanned vehicles is characterized in that a neural network is used to control the on-board system, which is trained directly on the input signals of the sensors and outputs the necessary control signals based on the learned patterns for this purpose, the architecture of the convolutional neural network directly matches the input raw image pixels with control commands, while the synthesis network is the basis of the model architecture and is responsible for extracting the characteristics of the environment, the second part is the path prediction network, which receives the feature vectors provided by the synthesis network and predicts the next few route points for the vehicle, then the output values from the path prediction network are fed to a proportional-integral-differentiative controller (PID controllers), which control the vehicle motion control systems, the synthesis network is divided into two branches: an image branch and a bird's eye view branch, combined through a transformer module, the image branch receives an RGB image as an input and uses a series of convolutional encoders to extract visual characteristics from the environment, the encoders work in different scales, capturing both fine details and contextual information, the resulting feature maps contain visual representations that capture the various patterns present in the image, the bird's eye view branch takes as input a point cloud display obtained from the lidar, which uses its own set of convolutional encoders, the information obtained from the lidar allows for geometric and spatial interpretation of the data, allowing the model to perceive a bird's eye view of the three-dimensional structure of the environment, the feature maps obtained from both branches are fed to the input of the transformer module, which performs joint processing of the data from the modality branches, while transformers are one of the deep learning models, and their architecture consists of encoder-decoder blocks and self-supervision layers, when processing information, the self-supervision mechanism allows the model to order the various elements according to importance weights, the transformer calculates an importance score for each element by comparing it with the others.

In a preferred embodiment, various types of sensors may be used in equipping the unmanned vehicle, including cameras, lidars, and radars.

BRIEF DESCRIPTION OF THE DRAWINGS

The essence of the invention is explained by the drawings, which show:

Fig.1 – architecture of the TransFuser model;

Fig.2 – auxiliary tasks used in TransFuser;

Fig.3 – input data of the KITTI dataset;

Fig.4 – KITTI semantic segmentation markup;

These drawings do not cover and, moreover, do not limit the entire scope of options for implementing this technical solution, but represent only illustrative material of a particular case of its implementation.

EMBODIMENT OF THE INVENTION

The claimed method of controlling on-board systems of unmanned vehicles using neural networks based on the transformer architecture is aimed at simplifying routine operations and ensuring safety for people. Accidents related to any type of transport occur mainly due to the human factor. Therefore, the development of unmanned vehicles that exclude the possibility of human error is becoming an increasingly important task.

The end-to-end learning method is one of the machine learning methods that deals with the transformation of input data into output data, while eliminating the need for manual feature processing. The method has shown high efficiency in such tasks as image classification and natural language processing. In the field of unmanned transport development, there is considerable interest in the application of end-to-end learning of neural networks of onboard systems, since it allows the model to directly learn from the input data and generate appropriate control commands.

Autonomous vehicles are an important area of research in the practical application of machine learning. The ability of unmanned vehicles to adequately perceive the environment is critical to their successful operation. To create an unmanned vehicle that can operate independently, without human intervention, it is necessary to equip the vehicle with a comprehensive sensor system. At the same time, it is also necessary to create a system for joint processing of sensor signals, for which a neural network is chosen in this paper. Below, we describe methods for solving the problem of joint processing of sensor signals using a neural network, and provide sources in which they are described.

Co-processing of sensor signals is the simultaneous processing of output signals from multiple sensors, including different types. In the context of unmanned vehicles, co-processing of sensor signals is one of the key safety components, allowing the on-board system to accurately perceive the environment and make automatic control decisions adequate to the situation.

There are different types of sensors that can be used in autonomous vehicles, including cameras, lidars, and radars. Each type of sensor has its own strengths and weaknesses, and combining multiple sensors can help overcome the problems of each one individually, thus providing a more accurate and reliable perception of the environment.

One widely used approach in joint processing of sensor signals is to use a probabilistic model such as a Bayesian filter. A Bayesian filter is a mathematical model that can combine information from multiple sensors and estimate the most probable state of the environment.

Other approaches to sensor fusion rely on deep learning neural networks of the type convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These methods are used to combine information from multiple sensors directly from raw signals.

Bayesian filters and deep learning methods, which have shown promising results in the field of autonomous vehicles, are the most common approaches in joint processing of sensor signals.

Algorithms and methods for joint processing of sensor signals are important in integrating the outputs of different sensors to obtain a more accurate description of the state of the environment. Table 1 lists some common methods for joint processing of sensor signals and their features.

Table 1. Common methods of joint processing of sensor signals and their features

Method Signs Advantages Kalman filter Linear model, Gaussian noise Optimal in linear systems with Gaussian noise Extended Kalman Filter Nonlinear model, Gaussian noise An extended version of the Kalman filter capable of handling nonlinear systems Neural networks Not a probabilistic model Can explore complex relationships between data Bayesian networks Probabilistic model Can take into account uncertainty and be used for argumentation

One of the main advantages of end-to-end learning for autonomous vehicles is that it reduces the reliance on human workers to manually label data and develop decision-making processes. This approach allows the model to learn directly from sensor inputs and output the necessary control signals based on the learned patterns. In addition, end-to-end learning models can be trained using large data sets, which helps improve the accuracy of the model and its generalization ability.

The stated method proposes a solution that introduces the concept of end-to-end learning in the field of autonomous vehicles. A convolutional neural network architecture is proposed that directly maps input raw image pixels to control commands without requiring manual data labeling. A large collection of ground vehicle driving videos was used in the training phase. The success of the proposed method was confirmed in real road conditions.

Architecture of the applied TransFuser model

TransFuser consists of two main parts: the first is a transformer-based synthesis network. The synthesis network, as shown in Figure 1, is the core architecture of the TransFuser model and is responsible for extracting environmental features. The second part is a path prediction network, which takes the feature vectors provided by the synthesis network and predicts the next few waypoints for the vehicle. The outputs from the path prediction network are then fed to PID controllers that control the vehicle's motion control systems.

The synthesis network is divided into two branches: an image branch and a bird's eye view (BEV) branch, which are combined via a transformer module. These components, in addition to the feature fusion process performed by the transformer module, are described in detail in the following sections.

1) Branches of modality:

The image branch takes an RGB image as input and uses a series of convolutional encoders to extract visual features from the environment. The encoders operate at different scales, capturing both fine detail and contextual information. The resulting feature maps contain visual representations that capture the various patterns present in the image.

The bird's eye view branch takes as input a point cloud image from the lidar. This branch uses its own set of convolutional encoders. The information obtained from the lidar enables geometric and spatial interpretation of the data, allowing the model to perceive a bird's eye view of the 3D structure of the environment.

2) Transformer module:

The feature maps obtained from both branches are fed to the input of the transformer module, which performs joint processing of the modality branches. Transformers are one of the deep learning models. Their architecture consists of encoder-decoder blocks and self-control layers, which are the distinctive feature of transformers.

When processing information, the self-checking mechanism allows the model to order the different elements according to importance weights. The transformer calculates an importance score for each element by comparing it with the others. The importance score also determines the contribution of all other elements to the representation of the element being evaluated.

The self-control mechanism is mathematically described as follows: integration of data from fuel and energy balances, regional gasification programs, as well as information on working with gas consumers into state information systems;

A sequence of input elements is given, defined as , where n is the length of the sequence. Each element is transformed into key ( K ), query ( Q ), and value ( V ) vectors using the learned linear projections:

Here , And – weight matrices. The self-checking mechanism calculates an estimate (α) for each pair of elements:

Where - query vector for the element , - key vector for the element , A - the dimension of the key vector. The scalar product is scaled taking into account , to reduce the influence of vector magnitudes.

The resulting estimates are then processed by the softmax function, which calculates the weights:

Finally, the self-supervision mechanism maps the attention weights to the value vectors to produce an output representation (AttOut) for each element:

The resulting output representations are then processed in the transformer model by feedforward neural networks.

The transformer's self-control mechanism has proven itself particularly well in unmanned ground transportation tasks. The model is able to capture complex relationships between objects and their environment, such as how the position and movement of a car relate to the positions of other vehicles, pedestrians, and obstacles.

3) Combining features:

In the process of combining features, two modalities are considered, each of which is represented by intermediate feature maps of dimension And To combine these modalities, they are concatenated, resulting in a tensor with the dimension:

The resulting tensor contains the input data for further processing and passes through the transformer module. The output of the module is a tensor of the same dimension, which is then transformed into two maps with dimensions corresponding to the original feature maps, And . These maps are used as inputs for subsequent steps in each modal branch. The final feature map from each branch undergoes additional processing. To reduce the dimensionality of the feature vectors, an averaging layer is used with a transformation to 512 dimensions, which allows preserving important information.

The main objective of this study is to evaluate the performance of TransFuser in perceiving the specificity of a new environment. Instead of integrating the feature maps of both modalities in the last layer, each map is fed to the corresponding transformer decoders for auxiliary tasks.

B. Auxiliary tasks

Auxiliary tasks in the context of the conducted research serve the purpose of improving the learning process by introducing additional signals or using regularization techniques, which ultimately leads to an increase in the performance of the main system. Moreover, they allow the network to implement more robust and generalizable representations.

In addition to the primary task of predicting waypoints, TransFuser uses four auxiliary loss functions that have shown great potential in previous studies involving control agents:

1) 2D Depth Prediction: Depth estimation refers to the process of predicting the distance of objects from a given viewpoint. TransFuser uses a method called 2D depth estimation, which aims to estimate depth based on two-dimensional images.

2) Semantic segmentation: The task is to label individual pixels in an image according to the specific semantic category they are associated with. This technique allows the model to understand the different objects and regions present in the image.

3) HD Map: In TransFuser, HD map is predicted using three-channel segmentation mask of bird's eye view. This HD map contains different classes such as lane markings, roads, providing details about passable and impassable areas.

4) Bounding Boxes. To detect additional vehicles, TransFuser uses the CenterNet decoder to estimate keypoints. It predicts vehicle locations from bird's eye view features using a convolutional decoder. Training involves creating a 2D target label with Gaussian kernels at the centers of objects. A two-step approach is used to solve vehicle orientation problems. First, TransFuser discretizes the relative rotation angle at each pixel. Then, a regression map is generated for vehicle size, position offsets, and orientation offsets. Training takes into account focusing loss, cross-entropy loss, and -losses, respectively, for position, regression and orientation mappings.

The original TransFuser model was initially trained on the CARLA simulator, a virtual environment for self-driving cars.

1) Real Vehicle Environment: The first step evaluates the performance of the model using a real vehicle dataset. This critical step aims to assess how effectively the model can interpret real data compared to synthesized data. While the CARLA simulator aims to provide a realistic environment, it is important to recognize the limitations in accurately representing the complexity of real-world scenes. For example, one limitation is the quality of the sensor data generated by CARLA, which can differ significantly from the output of real sensors. This discrepancy can manifest itself in changes in accuracy, noise, and potentially lead to significant changes in the behavior of the model.

The widely used KITTI Vision dataset was chosen for evaluation. The performance analysis compared the results of the TransFuser model with the performance of its peers presented in the KITTI ranking using standardized metrics. In particular, attention was focused on auxiliary tasks such as semantic segmentation, depth prediction, and bounding box prediction.

2) Simulation of the unmanned vehicle environment. The second stage evaluates the performance of the model in a simulator environment adapted for unmanned vessels. The pre-trained feature fusion module was finally tuned, after which the model was tested on auxiliary tasks. This stage aims to adapt the model to the characteristic features of the environment. The model was pre-tuned based on previously acquired information and is able to discern new features typical for the environment, while maintaining an understanding of the common features learned in the original terrestrial environment. The simulator data obtained from ship control scenarios on the Unreal Engine 4 platform were used. The model under study was evaluated in comparison with the classical tracking and joint processing of high-level signals, which is used here as a benchmark for performance comparison.

The previously used method for predicting the state of detected objects based on measurements and motion models uses an extended Kalman filter (EKF). The objects detected by the camera and lidar are processed jointly and compared. The objects detected by the camera are converted to 3D using post-processing, and the matching algorithm finds the center of the image, determines the sizes and classes of objects. The extended Kalman filter performs prediction taking into account the influence of data noise and measurement errors. The described tracking module allows for more accurate assessment of the state of objects and improves the perception of the environment by the on-board vehicle system.

3) Environment of a real motor transport vessel. At the final stage, the model is tested on real data obtained by onboard systems of unmanned vehicles. Similar to the initial stage, it is necessary to evaluate the performance of the model.

The datasets used in each stage of the experiments are examined taking into account characteristics such as input/output shape, ground truth labels, and metrics.

KITTI datasets consist of readings from various types of sensors commonly used in autonomous driving systems. The main sources of data in the set include:

- Camera images. The dataset provides black and white and color images captured by a high-resolution camera mounted on a vehicle. These images have a resolution of 1392 × 512 pixels.

- Lidar point clouds. The dataset includes 3D point cloud data captured by a Velodyne HDL-64E LiDAR sensor. Lidar points are represented as (x, y, z) coordinates in the vehicle coordinate system.

In addition to camera images and lidar point clouds, KITTI also provides GPS and IMU data, vehicle position, orientation, and speed information. However, in this example, only RGB images and lidar data are used as input to the model. Figure 3 shows an example of visualization of the input data of the KITTI dataset.

1) Benchmark data and metrics: The KITTI collection provides benchmark data for a range of tasks, including object tracking and detection, segmentation, and image depth estimation.

Image Depth Estimation. The dataset for this task is represented by depth maps, which provide pixel-level depth information. The main metrics used in evaluating this task are:

SILog Error:

SILog is a common metric in computer vision, especially in depth estimation tasks. It is the main metric used by the KITTI Vision Benchmark to rank models for depth estimation. SILog can be interpreted as the average logarithmic error per pixel between the predicted and true depth. Lower SILog values indicate better depth estimation.

iRMSE Deviation:

iRMSE is another metric used in the KITTI depth evaluation ranking. Similar to SILog, a lower iRMSE value generally indicates a better quality model.

Semantic segmentation of instances. The segmentation data is stored in each file in PNG image format, encoded as an 8-bit unsigned integer, with each pixel value representing its class. The KITTI data collection considers 34 classes, so the following approach is used to map them to the 7 classes considered by TransFuser:

1) All vehicle type classes from KITTI (e.g. car, truck, bus, train, etc.) are labeled as "vehicle".

2) The classes "road", "sidewalk" and "pedestrian" are displayed directly.

3) All other classes (including "lane markings" and "red light" from TransFuser) are considered "unlabeled" due to the lack of a corresponding label in the dataset.

To evaluate the semantic segmentation task, the average precision (AP) metric used by the KITTI benchmark is taken into account. For each class, the average precision over the domain is calculated, and then their average value is determined on a scale of overlap thresholds. This approach helps to smooth out any potential bias towards a specific value. The overlap threshold range includes values from 0.5 to 0.95 with a step of 0.05. As a result, 10 different overlap thresholds are considered during the evaluation.

In addition, in addition to average accuracy, additional metrics are also used, including pixel accuracy, precision, recall, and similarity. They are calculated as follows:

pixel precision

accuracy

completeness

similarity

The specified metrics allow for a comprehensive assessment of the model's performance.

Object Detection. KITTI contains a built-in benchmark for 3D object detection with vehicle and pedestrian bounding boxes. In this paper, KITTI is considered as a benchmark for evaluating the model's performance in the auxiliary bounding box problem.

The scoring process is based on calculating the percentage of detected bounding boxes that meet a minimum overlap threshold. For vehicles, 70% overlap is required, while for pedestrians, 50% overlap is sufficient. The resulting score is further divided into three different complexity levels, taking into account factors such as the minimum height of the bounding box, the degree of object overlap, and the maximum truncation. The TransFuser model focuses on predicting 2D bounding boxes from a bird's eye view perspective. To match the labels to the real values, a projection onto the XY plane is performed. This involves calculating the minimum and maximum value for each of the X and Y coordinates at all eight corners of the reference data bounding boxes. Based on these coordinates, four new points are created: ( ), ( ), ( ), ( ), which define the projected bounding parallelepiped.

B. Ship Simulator Data

For the second stage of experiments, we used data collected by us at the Center for Unmanned Technologies of Innopolis University. The provided data set consists of two scenarios, each containing about 500 frames. Each frame is described by five files:

1) A PNG file containing a 960×640 RGB image captured from the front camera.

2) XYZ file containing lidar point cloud data for a range of 150 meters in the format: X, Y, Z.

3) EXR file containing encoded image depth information as a 960×640 image.

4) A PNG file containing a 960×640 RGB image representing a semantic-level pixel segmentation using the color encoding of the camera image.

5) A TXT file containing a list of bounding boxes in the format: label, 8 corner points in XYZ space, rotation angle. Since the TransFuser model works with 2D bird's eye view bounding boxes, the bounding boxes were projected onto the XY plane using the same principle as the KITTI bounding boxes.

Thus, the TransFuser platform has demonstrated the ability to adapt to any environment, showing competitive performance compared to other known methods of joint processing of signals obtained from different sensors for unmanned vehicles.

Claims (2)

1. A method for controlling on-board systems of unmanned vehicles using neural networks based on a transformer architecture, including joint processing of output signals from sensors of the unmanned vehicle, characterized in that a neural network is used to control the on-board system, which is trained directly on the input signals of the sensors and outputs the necessary control signals based on the learned patterns, for this purpose the convolutional neural network architecture directly matches the input raw image pixels with control commands, wherein the synthesis network is the basis of the model architecture and is responsible for extracting environmental characteristics, the second part is a path prediction network that receives the feature vectors provided by the synthesis network and predicts the next several route points for the vehicle, then the output values from the path prediction network are fed to proportional-integral-differentiative controllers (PID controllers) that control the vehicle motion control systems, the synthesis network is divided into two branches: an image branch and a bird's eye view branch, combined through a transformer module, the branch the image branch takes as input an RGB image and uses a series of convolutional encoders to extract visual features from the environment, the encoders operate at different scales, capturing both fine detail and contextual information, the resulting feature maps contain visual representations that capture the various patterns present in the image, the bird's eye view branch takes as input a lidar point cloud display, which uses its own set of convolutional encoders, the information obtained from the lidar enables geometric and spatial interpretation of the data, allowing the model to perceive a bird's eye view of the 3D structure of the environment, the feature maps obtained from both branches are fed to the input of the transformer module, which jointly processes the data from the modality branches, while transformers are a type of deep learning model, and their architecture consists of encoder-decoder blocks and self-supervision layers, when processing information, the self-supervision mechanism allows the model to order the various elements according to importance weights, the transformer calculates an importance score for each element, comparing it with the others. 2. The control method according to paragraph 1, characterized in that the unmanned vehicle is equipped with cameras, lidars and radars.