RU2817534C1 - Method for automatic detection of objects using computer vision system installed on uav - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to methods of automatic detection of objects using a computer vision system installed on an unmanned aerial vehicle (UAV). Method includes steps of receiving frames from a video stream received from a camera of the UAV to a computing device, determining an offset of the current frame of the video stream relative to the previous processed frame in pixels, determining the size and position of the tracking window on the current frame, processing a fragment of the frame inside the tracking window by means of a neural network for detecting objects, determining the position of the objects detected in the fragment in the full frame. Further, determining the displacement in the frame of previously detected objects, which are inside the frame, but do not fall into the current tracking window by adding to the coordinates detected at the previous stages of processing objects, displacements in pixels between frames, and outputting the detection result.

EFFECT: high accuracy of detecting small-sized objects and objects.

6 cl, 4 dwg

Description

TECHNICAL FIELD

This technical solution relates to the field of technical vision and unmanned aerial vehicles (UAV), namely to a method for automatically detecting objects using a technical vision system installed on the UAV.

BACKGROUND OF THE ART

Traditional object detection methods using neural networks face difficulty in detecting small objects and objects located far in the frame, since they are represented by a small number of pixels in the image and have insufficient detail. There are known methods for increasing the accuracy of detecting small objects by recognizing each frame in parts, “running” a window over the frame. However, this method leads to a sharp decrease in the detection speed due to the increased computational costs of processing each frame.

Other ways to improve the accuracy of detecting small objects are to change the architecture of the neural network. For example, adding modules responsible for identifying features of small objects and removing those responsible for large objects. However, such an approach will require constant research and adaptation of the solution to new models.

The prior art contains information source WO 2022074643 A1, published on April 14, 2022, which discloses a method for georegistration using object identification based on machine learning. The system by which the method is carried out includes a video camera located on a vehicle, such as a UAV, which takes aerial photographs of the area. The success rate and accuracy of geosynchronization algorithms are improved by using a trained feedforward artificial neural network (ANN) to identify dynamic features that change over time in frames captured by a video camera. Such frames are marked, for example, by adding metadata. The tagged frames can be used in a geosynchronization algorithm, which can be based on comparison with reference images or can be based on a different or the same ANN, by removing the dynamic object from the frame or removing the tagged frame for the algorithm. A dynamic object may change over time due to environmental conditions such as weather changes or geographic changes.

The prior art contains information source US 20220028048 A1, published on January 27, 2022, which discloses a system and method for determining the position of an object using images obtained from a mobile image collection device. The method includes obtaining images of a target geographic area and telemetry information from an image acquisition means at the time of capture, analyzing each image to identify objects, and determining the position of objects. The method further includes determining an image capture height, determining an image position using the capture height and telemetry information, performing an image transformation based on the capture height and telemetry information, identifying objects in the transformed image, determining locations in the first pixels of the transformed image, and performing an inverse transformation. Transforming the first pixels to determine the locations of the second pixels in the image and determining the positions of objects in the area based on the locations of the second pixels in the captured image and the determined position of the image.

The proposed solution differs from the solutions known from the prior art in that in the proposed solution it is not necessary to scan each image frame, since information about objects is stored in several frames and it is enough to process one frame, and by adapting the size and position of the tracking window to the speed and height of movement UAVs increase the accuracy of detecting small objects.

SUMMARY OF THE INVENTION

The technical problem to be solved by the stated technical solution is the detection of small-sized objects and objects located far in the frame while maintaining the video stream processing speed at a level close to real time, as well as the absence of changes in the neural network architecture.

The technical result achieved by solving the above technical problem is to increase the accuracy of detection of small-sized objects and objects located far in the frame and maintain the video stream processing speed at a level close to real time by using the overlap of adjacent frames in the video stream. While maintaining the video stream processing speed at 15 or more frames per second, there is a high degree of overlap between adjacent frames. This means that information about objects is stored in several successive frames, taking into account the speed and direction of the optical flow, which depends on the speed, direction and altitude of the UAV flight; frames that have already been processed in previous frames do not need to be processed. By adapting the size and position of the tracking window to the speed and altitude of the UAV, the accuracy of detecting small objects is increased.

The claimed technical result is achieved by implementing a method for automatically detecting objects using a technical vision system installed on an unmanned aircraft (UAV), containing the following steps:

the computing device receives frames from the video stream received from the UAV camera;

determine the offset of the current frame of the video stream relative to the previous processed frame in pixels;

determine the size and position of the tracking window on the current frame;

process a frame fragment inside the tracking window using a neural network to detect objects;

determine the position of objects detected in the fragment in the full frame;

determine the displacement in the frame of previously detected objects that are located inside the frame, but do not fall into the current tracking window, by adding to the coordinates detected at the previous stages of object processing, a displacement in pixels between frames;

output the detection result.

In a particular embodiment of the proposed method, the offset of the current frame relative to the previous processed frame in pixels is carried out using an optical flow algorithm.

In a particular embodiment of the proposed method, the offset of the current frame relative to the previous processed frame in pixels is carried out using the block matching method.

In a particular embodiment of the proposed method, the offset of the current frame relative to the previous processed frame in pixels is carried out by determining the offset in pixels based on the height and speed of movement of the UAV and the orientation of the camera.

In a particular embodiment of the proposed method, the tracking window is located in the center of the frame.

In a particular embodiment of the proposed method, the tracking window moves over that part of the frame that will quickly move outside the frame.

DESCRIPTION OF DRAWINGS

The implementation of the invention will be described further in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

Figure 1 illustrates a diagram where frames missed due to the running time of the object detection algorithm by the neural network are not taken into account.

Fig. 2 illustrates a scheme for shifting a video stream along one axis at a constant speed to scan a frame using a tracking window.

Fig. 3 illustrates the scheme for shifting the video stream along two axes X and Y to scan a frame using a tracking window.

Fig. 4 illustrates an example of the implementation of the proposed solution.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description of the invention sets forth numerous implementation details designed to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used with or without these implementation details. In other cases, well-known methods, procedures and components have not been described in detail so as not to obscure the features of the present invention.

In addition, from the above discussion it will be clear that the invention is not limited to the above implementation. Numerous possible modifications, alterations, variations and substitutions, while retaining the spirit and form of the present invention, will be apparent to those skilled in the art.

The proposed method is carried out through the operation of a system consisting of a UAV and a computing device.

The computing device receives frames from the video stream received from the UAV camera. The BVS camera receives an image using optical lenses and an image matrix, which breaks the image into individual pixels. The resulting frame is an electronic image in digital format, that is, a set of numbers that correspond to the brightness of each pixel in the image. Frames are received in real time. At this step, the last available decoded frame is taken from the video stream.

Next, the offset of the current frame of the video stream relative to the previous processed frame is determined in pixels. The previous frame that was processed by this algorithm is used. Missed frames, if there are any due to the operating time of the object detection algorithm by the neural network, are not taken into account (Fig. 1).

Various algorithms can be used to determine the offset between frames in a video stream.

One of the most common algorithms is the optical flow algorithm, which allows you to determine displacement vectors for each pixel in the image. The optical flow algorithm is based on the assumption that the brightness of each point in the image is maintained between adjacent frames. It uses the calculation of the luminance gradient of an image and applies optimization techniques such as the Lucas-Kanade method to find pixel displacement vectors between two frames.

Another method for determining offset between frames is the block matching method, which compares blocks of pixels on the current frame with corresponding blocks on the previous frame. As a result of the comparison, displacement vectors for each block are determined.

The third method is to calculate the displacement in pixels based on the height and speed of the UAV. To calculate the pixel offset based on the height and speed of the UAV, you can use the following formula:

dx=V * time_interval * frame_resolution_x / (2 * N * tan(A))

dy=V * time_interval * frame_resolution_y / (2 * H * tan(A))

Where:

dx - offset between frames along the x axis in pixels;

dy - offset between frames along the y-axis in pixels;

H - UAV flight altitude above the surface;

V is the speed of the UAV;

A - camera viewing angle;

time_interval - time interval between frames (update time), which can be defined as the inverse value of the frame rate;

frame_resolution_x - frame resolution in pixels along the x axis;

frame_resolution_y - frame resolution in pixels along the y-axis.

To use this formula, you need to know the altitude and speed of the UAV; these parameters are obtained from the UAV flight controller, as well as the camera viewing angle. Camera viewing angle determines the camera's field of view and therefore the number of pixels per degree of viewing angle. The camera should be pointed straight down.

Next, the size and position of the tracking window on the current frame are determined. The effective size of the tracking window depends on the architecture of the neural network used for object detection (the presence of modules responsible for identifying features of small objects), the size of objects in the training dataset, and the size of potential objects in pixels. In this case, the minimum window size depends on the frame shift speed in order to have time to process the entire image space inside the frame before it moves outside the frame.

The tracking window can always be located in the center of the frame (or any other place) and increase the accuracy of detecting objects whose image falls in the area in the center of the frame. In this case, it is necessary to plan the shooting in such a way that, while moving the camera, the central area of the frame captures the entire space of interest, also taking into account the movement of the objects being recognized.

Another way is to sequentially “scan” the frame using a tracking window. In the degenerate case, when the camera is directed straight down and rotated in the direction of motion of the UAV, the displacement of the video stream can be represented as a displacement along one axis at a constant speed (Fig. 2). The tracking window has the shape of a square with width W _win . The square size is chosen in accordance with the proportions of the input layers of the neural networks used for object detection. Frame width - W. Offset between adjacent frames - Δ _pix .

The number of iterations for scanning with a tracking window across the entire frame width is determined by:

In order to have time to scan with a tracking window across the entire width of the frame before this area moves outside the frame, the displacement rate must be less than the speed of the tracking window width divided by the number of width scanning iterations, rounded up:

The minimum width of the tracking window can be calculated using the formula:

In other words, the area of the tracking window must be no less than the area of the part of the frame that will completely move outside the frame on the next frame.

In the method under consideration, the tracking window moves over that part of the frame that will quickly move outside the frame. Any algorithm can be used to move the tracking window, for example, from left to right, or from the center to the edges.

When the video stream is shifted along two axes X and Y (Fig. 3), the minimum window size is also determined by the area of the frame, which will completely shift outside the frame on the next frame. The choice of the position of the tracking window in the frame is determined by the residual area, consisting of the part of the frame that will move outside the frame minus the area covered by the tracking window in the previous frame, S _w and S _h in Fig. 3. On which side the residual area is larger, the next tracking window is placed on that side. If Sw>Sh, then a new tracking window is placed to the right of the last tracking window in the horizontal row. If Sh<Sw then it is placed on top of the last tracking window in the vertical row

Next, the frame fragment inside the tracking window is processed using a neural network to detect objects.

The fragment is transmitted to the neural network in accordance with the size of the input layer. If necessary, the fragment is converted to the required size. For example, the input layer has a matrix size of 1280x1280x3, and a fragment size of 640x640x3 (an RGB image with three color channels). In this case, the fragment is scaled to the size of the input layer of the neural network. For scaling, any of the popular algorithms can be used, for example, the linear interpolation method. Scaling is possible both with and without saving proportions. If the proportions do not match the size of the input layer of the neural network, the space outside the image can be filled with zeros (meaning no brightness).

To detect objects inside the tracking window, any neural network designed to solve the object detection problem, which consists in determining the coordinates of an object in the image and the class of the object, can be used. For example, YOLO, SSD, Retina Net and others. The result of image processing by a neural network is an array containing the relative coordinates of the object's center, width and height, object class, and the neural network's confidence for each object. To detect objects of different scales, neural networks use pyramidal image processing when the image scale changes during training. Standard neural network models are designed to detect objects of different sizes. Therefore, when enlarging an object by selecting a fragment of an image, instead of feeding the entire image to the input of the neural network, the neural network better detects signs of a small-sized object, since the size of the object in this case is more consistent with the scale of the objects on which the neural network was trained.

Next, the position of the objects detected in the fragment in the full frame is determined. The position is determined by the coordinates in pixels inside the fragment, shifted by the values of the fragment’s position inside the frame. To do this, the absolute coordinates of the fragment inside the frame X _fr and Y _fr are added to the absolute coordinates X ₁ and Y _fr of the object inside the fragment. Thus, the global coordinates of the object inside the frame will be X _hl =X ₁ +X _fr , Y _hl =Y ₁ +Y _fr .

Next, the displacement in the frame of previously detected objects that are inside the frame, but do not fall into the current tracking window, is determined using data (flight altitude, UAV speed) received from the UAV flight controller by adding to the coordinates detected at the previous processing stages objects, displacement in pixels between frames. To do this, a displacement in pixels between frames dx and dy is added to the absolute coordinates of the object X and Y. Next, it is determined whether the object is still in the frame. If the object is outside the frame, the calculation of its position in the next frames stops. Thus, the new coordinates of the object inside the frame will be:

X _new =X+dх, subject to 0<X<X _max

Y _new =Y+dy, subject to 0<Y<Y _max

Based on the processing result, the detection result is output. Detection results are transmitted in the format [x, y, w, h, class] of each object in the frame, where x, y, w, h are the relative coordinates of the object’s center, its width and height. Class - denotes the index of the object's class. Additionally, the neural network's confidence in the class value can be passed. The output is provided in the form of the desired display of detected objects. This can be a visual display in the form of frames and class names directly in the video stream, or transfer of results to external algorithms for further processing.

Next, with reference to FIG. 4, an example of the invention is described. A quadcopter-type UAV is used, based on the PX4 autopilot, the camera (1) transmits a video stream via a Wi-Fi wireless communication channel, the codec is H.264, the flight controller (4) transmits information about the position of the UAV via the MAVLink protocol through the same communication channel, computing the device represents a personal computer with the Windows 10 operating system, frame capture (2) is performed using the OpenCV library, calculation of the position of the tracking window (3) as a software function, processing by a neural network (5) YOLOv5 for object detection, calculation of the position of objects as a software function functions.

According to the scheme, the video stream from the camera (1) is transmitted to a personal computer, where a frame is captured using the OpenCV library (2). Then, using an algorithm to calculate the position of the tracking window (3), the position of the tracking window in the frame is calculated. Image processing for object detection is performed using the YOLOv5 neural network (5). The UAV coordinates (4) received from the flight controller are used to calculate the displacement of previously detected objects that were not included in the current tracking window using an algorithm for calculating the position of detected objects (6). The processing results are transferred to the algorithm for calculating the position of detected objects (6), which calculates the position of objects in the image. Based on the processing result, the detection result is output.

To determine the position of the UAV, a flight controller (4) is used, which transmits information about its position via the MAVLink protocol via a Wi-Fi wireless communication channel. Calculations are performed on a personal computer, which can be any PC with the Windows 10 operating system installed and a discrete video card supporting CUDA computing cores.

A computing system that provides data processing necessary to implement the claimed solution generally contains components such as: one or more processors, at least one memory, data storage means, input/output interfaces, input means, network communication means. When executing machine-readable instructions contained in the RAM, the device processor is configured to perform basic computing operations necessary for the operation of the device or the functionality of one or more of its components. Memory, as a rule, is made in the form of RAM, into which the necessary program logic is loaded to provide the required functionality. When implementing the proposed solution, the amount of memory required to implement the proposed solution is allocated. Data storage media can be in the form of HDDs, SSD drives, raid arrays, network storage, flash memory, etc. The tool allows long-term storage of various types of information. Interfaces are standard means for connecting and operating peripheral and other devices, for example, USB, RS232, RJ45, COM, HDMI, PS/2, Lightning, etc. As data input means in any embodiment of a system that implements the described method, a keyboard, joystick, display (touch display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc. can be used. Network interaction means are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. Using the tools, the organization of data exchange is ensured via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM. The device components are interconnected via a common data bus.

In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims (13)

1. A method for automatically detecting objects using a technical vision system installed on an unmanned aircraft (UAV), containing the following steps: the computing device receives frames from the video stream received from the UAV camera; determine the offset of the current frame of the video stream relative to the previous processed frame in pixels; determine the size and position of the tracking window on the current frame; process a frame fragment inside the tracking window using a neural network to detect objects; determine the position of objects detected in the fragment in the full frame; determine the displacement in the frame of previously detected objects that are located inside the frame, but do not fall into the current tracking window, by adding to the coordinates detected at the previous stages of object processing, a displacement in pixels between frames; output the detection result. 2. The method according to claim 1, characterized in that the offset of the current frame relative to the previous processed frame in pixels is carried out using an optical flow algorithm. 3. The method according to claim 1, characterized in that the offset of the current frame relative to the previous processed frame in pixels is carried out using the block matching method. 4. The method according to claim 1, characterized in that the offset of the current frame relative to the previous processed frame in pixels is carried out by determining the offset in pixels based on the height and speed of movement of the UAV and the orientation of the camera. 5. The method according to claim 1, characterized in that the tracking window is located in the center of the frame. 6. The method according to claim 1, characterized in that the tracking window moves over that part of the frame that will quickly move outside the frame.